Compare commits

...

18 Commits

Author SHA1 Message Date
sinohqb
a665b496b0 chore(v0.9): wrap over-length lines and record spec rulings
Some checks failed
CI / test (push) Failing after 39s
Wrap the judge prompt and two docstrings past the 120-col convention;
record three implementation rulings in the v0.9 spec (exploration read
outlets, round-based sampling, findings carrying all ratings).
2026-08-04 02:52:45 +08:00
sinohqb
b26c432f3a refactor(frontend): reuse ChatBubble for exploration drill-down
The drawer hand-rolled chat bubbles duplicating ChatBubble's shape.
Add a mirrored layout prop (virtual user left, target right) and reuse
the shared component.
2026-08-04 02:50:26 +08:00
sinohqb
ef4c094082 refactor(exploration): share one fetch+aggregate helper across outlets
report, markdown and analysis each repeated the
summarize_exploration(repo.list_by_campaign(...)) shape; collapse it
into summarize_campaign_exploration so the aggregation has one home.
2026-08-04 02:48:59 +08:00
sinohqb
936640fb36 fix(exploration): include all judge findings instead of poor-only
Ticket 05 asks the judge review conclusions to flow into the report
verbatim; the aggregation silently dropped good/acceptable dimensions.
Collect every finding sorted poor-first and color drawer tags by rating.
2026-08-04 02:45:41 +08:00
sinohqb
7f68afd765 docs(v0.9): mark ticket 06 done after Playwright browser verification 2026-08-04 02:27:37 +08:00
sinohqb
9abf572949 fix(exploration): flatten dict reply payloads instead of storing str(dict)
E2E on t480 showed assistant bubbles rendering {'content': '...'} because
tutu replies carry msgBody as a parsed object and the router stored
str(reply.content). Coerce to the inner text before persisting.
2026-08-04 02:26:24 +08:00
sinohqb
958cefc380 docs(v0.9): mark ticket 08 done after t480 cron registration and E2E 2026-08-04 02:09:36 +08:00
sinohqb
140bb1eb89 docs(v0.9): mark ticket 08 skill deliverables done pending t480 ops 2026-08-04 01:32:38 +08:00
sinohqb
fc167a8e4e feat(openclaw): patrol/exploration skill with loop-based skill sync 2026-08-04 01:32:13 +08:00
sinohqb
3c97e6e1ba docs(v0.9): mark ticket 07 finalize settlement done 2026-08-04 01:24:00 +08:00
sinohqb
62f7031b5b feat(exploration): expire dangling sessions on campaign finalize 2026-08-04 01:23:37 +08:00
sinohqb
b21dce73c5 docs(v0.9): mark ticket 06 code complete pending browser verification 2026-08-04 00:25:12 +08:00
sinohqb
a12eb09da3 feat(exploration): report drawer exploration findings section with session drill-down 2026-08-04 00:24:45 +08:00
sinohqb
51980016a4 docs(v0.9): mark ticket 05 exploration findings flow done 2026-08-03 19:16:47 +08:00
sinohqb
2484c207af feat(exploration): findings flow into report, analysis and export
Exploration sessions aggregate into a single exploration summary
(session counts, goal-achievement rate, issue lists from experience
records, judge conclusions when reviewed) that feeds three exits:
the campaign report gains an exploration dimension, the v0.7 analysis
stage-two input gains the summary (stats only, never full dialogues),
and the Markdown export appends a findings appendix after analysis and
comparison. With no exploration data every output stays unchanged.
2026-08-03 19:16:33 +08:00
sinohqb
9cf64ab0e2 test(report): aggregation edge cases and output format validation
Adds unit tests for the campaign report beyond value correctness:
cancellation exclusion from denominators (ADR-0004), window clamping,
naive/aware timestamp equivalence, plus schema validation of the report
dict (key sets, types, rate bounds, contiguous buckets, capability sort,
ISO-UTC timestamps) and consistency with the single aggregate_runs seam.
2026-08-03 19:03:44 +08:00
sinohqb
8ed4221b0f docs(v0.9): mark ticket 04 judge sampling review done 2026-08-03 18:59:53 +08:00
sinohqb
2285a25009 feat(exploration): judge sampling review after session close
After an exploration session closes, the platform samples up to 3
conversation rounds and runs an independent judge-role review through
the v0.7 ChatClient seam, persisting quality-dimension conclusions
(attitude, professionalism, hallucination) into the session's
judge_review. The review runs as a background task: failures are
recorded without touching session state or the first-hand experience
record, and a missing model config skips silently.
2026-08-03 18:59:40 +08:00
29 changed files with 1967 additions and 58 deletions

View File

@ -4,9 +4,9 @@
**Blocked by:** 01会话关闭事件与对话数据
**Status:** ready-for-agent
**Status:** completed
- [ ] 会话关闭后自动触发抽样复核,结果结构化落库
- [ ] 抽样上限生效;无模型配置时静默跳过
- [ ] 复核失败落错误,会话仍为 completed
- [ ] 假客户端测试覆盖编排、解析失败、无模型分支先例v0.7 分析测试)
- [x] 会话关闭后自动触发抽样复核,结果结构化落库
- [x] 抽样上限生效;无模型配置时静默跳过
- [x] 复核失败落错误,会话仍为 completed
- [x] 假客户端测试覆盖编排、解析失败、无模型分支先例v0.7 分析测试)

View File

@ -4,9 +4,9 @@
**Blocked by:** 01聚合主料是体验记录04 若已就位则复核结论一并纳入,未就位不阻塞
**Status:** ready-for-agent
**Status:** completed
- [ ] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
- [ ] 分析输入含探索摘要且不含全量对话
- [ ] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
- [ ] 纯函数渲染与聚合单测 + 导出集成测试先例report_render 单测、活动导出测试)
- [x] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
- [x] 分析输入含探索摘要且不含全量对话
- [x] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
- [x] 纯函数渲染与聚合单测 + 导出集成测试先例report_render 单测、活动导出测试)

View File

@ -4,9 +4,9 @@
**Blocked by:** 05报告聚合与查询端点就绪
**Status:** ready-for-agent
**Status:** completeda12eb09 + 9abf572 回复文本修复Playwright 浏览器验证通过)
- [ ] 报告抽屉呈现探索发现区块,数据与后端口径一致
- [ ] 会话详情展示完整对话(角色、内容、延迟)与体验记录
- [ ] 无探索数据时区块隐藏
- [ ] tsc 通过;人工在浏览器验证展开/下钻交互
- [x] 报告抽屉呈现探索发现区块,数据与后端口径一致
- [x] 会话详情展示完整对话(角色、内容、延迟)与体验记录
- [x] 无探索数据时区块隐藏
- [x] tsc 通过;浏览器验证展开/下钻交互Playwright 在 t480 实测区块渲染、问题清单、judge 复核、会话下钻气泡与体验卡片均正常,截图 /tmp/ticket06

View File

@ -4,9 +4,9 @@
**Blocked by:** 01会话实体与状态机
**Status:** ready-for-agent
**Status:** completed
- [ ] 活动终态结算时 running 会话转 expired后续消息 409
- [ ] 加速线自动触发被拒并附原因;手动触发可用
- [ ] 正式线两种触发来源均可用
- [ ] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)
- [x] 活动终态结算时 running 会话转 expired后续消息 409
- [x] 加速线自动触发被拒并附原因;手动触发可用
- [x] 正式线两种触发来源均可用
- [x] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)

View File

@ -4,9 +4,9 @@
**Blocked by:** 03巡检 API 契约就绪才能写说明书)
**Status:** ready-for-agent
**Status:** completedskill fc167a8常驻作业 e6f19ac8 已注册并重启存活E2E 活动 e1f3bdde 自动派发会话 3e46b6a3体验记录落库活动已收尾取消
- [ ] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
- [ ] 部署同步管线分发 skill复用现有同步步骤
- [ ] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
- [ ] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库
- [x] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
- [x] 部署同步管线分发 skill复用现有同步步骤
- [x] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
- [x] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库

View File

@ -51,7 +51,10 @@
- `POST /api/exploration/sessions`body = campaign_id、persona、goal、seed_ref可空平台硬校验活动存在且 running、正式线仅接受 auto 或 manual 触发;加速线仅接受 manual预算余量→ 创建 running 会话;超限 409 + 原因
- `POST /api/exploration/sessions/{id}/messages`body = content平台转发到目标的通道复用 ChannelFactory、持久化双方轮次、返回回复与延迟单会话轮数超限 409会话非 running 状态 409
- `POST /api/exploration/sessions/{id}/close`body = 体验记录goal_achieved: bool、blockers[]、misled[]、emotion、notes结构校验 + 非法值归一(沿 v0.7 白名单经验);会话转 completed
- `GET /api/campaigns/{id}/exploration`:会话列表 + 探索发现聚合(会话数、达成率、问题清单)
- 探索发现读出口(实现裁决:不设独立 `/api/campaigns/{id}/exploration`,避免与报告链重复聚合):
- `GET /api/campaigns/{id}/report``exploration` 键携带探索发现聚合会话数、达成率、问题清单、judge 复核),无探索数据时缺则无痕
- `GET /api/exploration/campaigns/{id}/sessions`:会话列表(下钻用)
- `GET /api/exploration/sessions/{id}/messages`:单会话对话记录(下钻用)
- 活动窗口 finalize 时:仍 running 的会话转 expired不再接受消息`resolve_finalize` 同处挂接)
### 护栏(平台硬执行,不信任客户端自律)
@ -62,11 +65,11 @@
### 判定双证据线
- 体验判定close 接口收结构化自报,为第一手证据
- judge 抽样复核:会话结束后平台对对话抽样(默认 ≤3 段,控 token经 judge 岗位模型产出质量维度复核ChatClient 可注入(沿 v0.7 分析 seam异步后台执行失败落错误不阻塞
- judge 抽样复核:会话结束后平台对对话抽样(实现裁决:按「轮」抽样,默认 ≤3 轮即最多 6 条消息、每条截断 500 字符控 token经 judge 岗位模型产出质量维度复核ChatClient 可注入(沿 v0.7 分析 seam异步后台执行失败落错误不阻塞
### 报告 / 分析 / 导出
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);周期对比口径不变(探索数据不参与)
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);judge 复核结论一并纳入实现裁决findings 全量收各档发现、poor 档排前,而非只收 poor周期对比口径不变(探索数据不参与)
- v0.7 分析输入追加探索**摘要**(问题清单 + 达成统计,非全量对话)
- Markdown 导出:存在探索数据时追加「探索发现」附录(缺则无痕),附录顺序:智能分析 → 周期对比 → 探索发现

View File

@ -14,6 +14,7 @@ from typing import Any, Awaitable, Callable, Optional
from sqlmodel import Session
from agenteval.evaluation.report import generate_campaign_report
from agenteval.exploration.summary import summarize_campaign_exploration
from agenteval.model_gateway import ModelGateway
from agenteval.models import Campaign, ModelCapability, RunStatus
from agenteval.services.model_configs import (
@ -152,6 +153,7 @@ async def _synthesize(
report: dict[str, Any],
stage1: list[dict[str, Any]],
chat_client: ChatClient,
exploration_summary: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
"""阶段二:汇总各场景产出,产总体结论 + 跨场景问题 + 优先级建议。"""
system_prompt = (
@ -163,17 +165,18 @@ async def _synthesize(
'"suggestions": [{"priority": 1, "text": "可执行的改善建议"}]}'
";问题按严重度从高到低排列,建议按优先级排列。全部使用中文。"
)
user_prompt = json.dumps(
{
"活动": {
"名称": campaign.name,
"窗口秒数": campaign.window_seconds,
"总体指标": report.get("summary", {}),
},
"各场景诊断": stage1,
payload: dict[str, Any] = {
"活动": {
"名称": campaign.name,
"窗口秒数": campaign.window_seconds,
"总体指标": report.get("summary", {}),
},
ensure_ascii=False,
)
"各场景诊断": stage1,
}
if exploration_summary is not None:
# 探索式评测证据线:只给统计与问题清单,不含全量对话
payload["探索发现"] = exploration_summary
user_prompt = json.dumps(payload, ensure_ascii=False)
parsed = _parse_stage(
await chat_client([
{"role": "system", "content": system_prompt},
@ -194,6 +197,7 @@ async def analyze_campaign(
failure_samples: dict[str, list[dict[str, str]]],
valid_run_ids: set[str],
chat_client: ChatClient,
exploration_summary: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
"""两阶段编排:阶段一按场景并行诊断,阶段二综合研判。
@ -209,7 +213,7 @@ async def analyze_campaign(
_analyze_scenario(entry, failure_samples.get(entry["scenario_id"], []), chat_client)
for entry in capability
])
stage2 = await _synthesize(campaign, report, list(stage1), chat_client)
stage2 = await _synthesize(campaign, report, list(stage1), chat_client, exploration_summary=exploration_summary)
valid_scenario_ids = {entry["scenario_id"] for entry in capability}
problems = []
@ -310,6 +314,7 @@ async def execute_campaign_analysis(
failure_samples=collect_failure_samples(campaign_id, session),
valid_run_ids={r.id for r in runs if r.id},
chat_client=client,
exploration_summary=summarize_campaign_exploration(session, campaign_id),
)
except Exception as exc:
_logger.warning("活动 %s 智能分析失败: %s", campaign_id, exc)

View File

@ -30,6 +30,7 @@ from agenteval.evaluation.campaign_scheduler import (
resolve_finalize,
)
from agenteval.evaluation.engine import EvalEngine
from agenteval.exploration.settlement import settle_campaign_sessions
from agenteval.models import (
Campaign,
CampaignStatus,
@ -201,6 +202,18 @@ def _auto_start_analysis(campaign: Campaign, session: Session) -> None:
_logger.warning("活动 %s 自动分析触发失败(已跳过): %s", campaign.id, exc)
def _settle_exploration(campaign_id: str, session: Session) -> None:
"""Expire dangling exploration sessions on completion.
Never blocks completion: a settlement failure is logged and skipped, the
same non-blocking semantics as ``_auto_start_analysis``.
"""
try:
settle_campaign_sessions(campaign_id, session)
except Exception as exc:
_logger.warning("活动 %s 探索会话结算失败(已跳过): %s", campaign_id, exc)
# ── durable scheduler loop ──────────────────────────────────────────────────
@ -250,6 +263,7 @@ async def run_campaign_loop(
current.status = CampaignStatus.COMPLETED
current.completed_at = utc_now()
repo.update(current)
_settle_exploration(campaign_id, session)
_auto_start_analysis(current, session)
return

View File

@ -347,19 +347,63 @@ def _window_line(report: dict[str, Any]) -> str:
return f"**窗口**: {_offset(window)}(加速调试线 ×{scale:g}{wall}"
def _render_exploration_lines(exploration: dict[str, Any]) -> list[str]:
"""探索发现附录:会话统计 + 问题清单 + judge 复核发现(有才渲染)。"""
lines: list[str] = [
"",
"## 探索发现",
"",
"| 指标 | 数值 |",
"|------|------|",
f"| 探索会话数 | {exploration.get('session_count', 0)} |",
f"| 有体验记录会话数 | {exploration.get('sessions_with_experience', 0)} |",
f"| 目标达成率 | {_pct(exploration.get('goal_achievement_rate'))} |",
"",
"### 问题清单",
"",
]
issues = exploration.get("issues") or []
misled = exploration.get("misled") or []
if not issues and not misled:
lines.append("")
else:
for item in issues:
lines.append(f"- {item['issue']} ×{item['count']}")
for item in misled:
lines.append(f"- (被误导){item['issue']} ×{item['count']}")
judge = exploration.get("judge_review")
if judge:
lines += [
"",
"### judge 复核",
"",
f"已复核 {judge.get('reviewed_sessions', 0)} 个会话:",
]
findings = judge.get("findings") or []
if findings:
for item in findings:
lines.append(f"- [{item.get('dimension')}/{item.get('rating')}] {item.get('comment')}")
else:
lines.append("- 未发现问题")
for summary_text in judge.get("summaries") or []:
lines.append(f"- 复核结论:{summary_text}")
return lines
def render_campaign_markdown(
report: dict[str, Any],
*,
analysis: Optional[dict[str, Any]] = None,
comparison: Optional[dict[str, Any]] = None,
exploration: Optional[dict[str, Any]] = None,
target_name: Optional[str] = None,
scenario_names: Optional[dict[str, str]] = None,
) -> str:
"""Render a dual-axis campaign report dict as Markdown.
``analysis`` and ``comparison`` are the stored 智能分析 / 周期对比 results
(completed only); when absent the corresponding appendix is omitted
entirely.
(completed only); ``exploration`` is the on-the-fly 探索发现 aggregate;
when absent the corresponding appendix is omitted entirely (缺则无痕).
"""
s = report["summary"]
status = _CAMPAIGN_STATUS_LABELS.get(report.get("status"), report.get("status") or "-")
@ -409,4 +453,6 @@ def render_campaign_markdown(
lines += _render_analysis_lines(analysis, scenario_names or {})
if comparison:
lines += _render_comparison_lines(comparison, scenario_names or {})
if exploration:
lines += _render_exploration_lines(exploration)
return "\n".join(lines)

View File

@ -0,0 +1,192 @@
"""Judge sampling review for exploration sessions (judge 岗位抽样复核).
会话关闭后平台对该会话对话抽样默认 3 token judge 岗位模型
独立复核产出质量维度结论态度专业性幻觉落入会话的 ``judge_review``
复核是异步后台执行失败落错误不阻塞会话状态也不影响体验记录这条第一手
证据线未配置模型时静默跳过LLM 调用沿 v0.7 分析的 ``ChatClient`` 接缝
注入测试用假客户端覆盖
"""
import asyncio
import json
import logging
from typing import Any, Optional
from sqlmodel import Session
from agenteval.evaluation.analysis import (
ChatClient,
gateway_chat_client,
resolve_analysis_model,
)
from agenteval.exploration.models import ExplorationMessage, ExplorationSession
from agenteval.models import Campaign
from agenteval.services.model_configs import ModelRuntimeConfig
from agenteval.storage.db import get_session, iso_utc, utc_now
from agenteval.storage.repository import (
CampaignRepository,
ExplorationMessageRepository,
ExplorationSessionRepository,
)
from agenteval.utils.llm import parse_json_from_llm_text
_logger = logging.getLogger("agenteval")
MAX_JUDGE_SAMPLES = 3
SAMPLE_TEXT_LIMIT = 500
VALID_DIMENSIONS = ("attitude", "professionalism", "hallucination")
VALID_RATINGS = ("good", "acceptable", "poor")
JUDGE_SYSTEM_PROMPT = """你是评测平台的 judge 岗位模型,负责独立复核一段「虚拟用户」与被评对象的探索对话。
只依据给定对话抽样判断不要臆测抽样之外的内容
输出必须是合法 JSON且仅包含以下结构
{
"dimensions": [
{
"dimension": "attitude | professionalism | hallucination",
"rating": "good | acceptable | poor",
"comment": "一句话依据"
}
],
"summary": "一句话总体结论"
}
维度说明attitude=服务态度professionalism=专业性流程与答复正确性
hallucination=幻觉编造事实政策或能力无幻觉时 rating good
三个维度必须各出现一次"""
class JudgeReviewError(RuntimeError):
"""judge 复核失败(模型输出无法解析或调用失败),不影响会话状态。"""
def resolve_judge_model(campaign: Campaign, session: Session) -> Optional[ModelRuntimeConfig]:
"""解析 judge 岗位模型:沿用活动分析模型口径(活动覆盖 ?? 全局分析默认)。"""
return resolve_analysis_model(campaign, session)
def sample_round_indexes(rounds: list[int], limit: int = MAX_JUDGE_SAMPLES) -> list[int]:
"""超限时均匀取样(含首尾),保证抽样可复现。"""
if len(rounds) <= limit:
return list(rounds)
span = len(rounds) - 1
return [rounds[round(i * span / (limit - 1))] for i in range(limit)]
def build_judge_messages(
session_obj: ExplorationSession, samples: list[ExplorationMessage]
) -> list[dict[str, str]]:
transcript = []
for message in samples:
speaker = "虚拟用户" if message.role == "user" else "被评对象"
content = str(message.content)[:SAMPLE_TEXT_LIMIT]
transcript.append(f"[第 {message.round_index} 轮] {speaker}: {content}")
payload = {
"persona": session_obj.persona,
"goal": session_obj.goal,
"transcript": transcript,
}
return [
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
]
def normalize_judge_review(raw: Any) -> dict[str, Any]:
"""白名单归一:未知维度丢弃,非法档位归 acceptable沿 v0.7 白名单经验)。"""
dims = raw.get("dimensions") if isinstance(raw, dict) else None
dims = dims if isinstance(dims, list) else []
normalized = []
for item in dims:
if not isinstance(item, dict):
continue
name = item.get("dimension")
if name not in VALID_DIMENSIONS:
continue
rating = item.get("rating")
normalized.append(
{
"dimension": name,
"rating": rating if rating in VALID_RATINGS else "acceptable",
"comment": str(item.get("comment") or "")[:SAMPLE_TEXT_LIMIT],
}
)
summary = str(raw.get("summary") or "")[:SAMPLE_TEXT_LIMIT] if isinstance(raw, dict) else ""
return {"dimensions": normalized, "summary": summary}
async def judge_conversation(
session_obj: ExplorationSession,
samples: list[ExplorationMessage],
*,
chat_client: ChatClient,
) -> dict[str, Any]:
text = await chat_client(build_judge_messages(session_obj, samples))
try:
raw = parse_json_from_llm_text(text)
except Exception as exc:
raise JudgeReviewError(f"judge 输出解析失败: {exc}") from exc
if not isinstance(raw, dict):
raise JudgeReviewError("judge 输出解析失败: 输出不是 JSON 对象")
review = normalize_judge_review(raw)
review["sampled_rounds"] = sorted({m.round_index for m in samples})
return review
async def execute_judge_review(
exploration_session_id: str,
*,
chat_client: Optional[ChatClient] = None,
) -> None:
"""后台执行体:抽样 → judge 复核 → judge_review 落库。
Runs 同款后台任务约定自持 Sessiontry/finally 关闭失败落 error
不阻塞会话状态未配置模型时静默跳过
"""
session = get_session()
try:
repo = ExplorationSessionRepository(session)
session_obj = repo.get(exploration_session_id)
if not session_obj:
return
campaign = CampaignRepository(session).get(session_obj.campaign_id)
runtime = resolve_judge_model(campaign, session) if campaign else None
if runtime is None:
_logger.info("探索会话 %s 跳过 judge 复核:未配置模型", exploration_session_id)
return
messages = ExplorationMessageRepository(session).list_by_session(session_obj.id)
round_indexes = sorted({m.round_index for m in messages})
sampled = set(sample_round_indexes(round_indexes))
samples = [m for m in messages if m.round_index in sampled]
if not samples:
return
client = chat_client or gateway_chat_client(runtime)
reviewed_at = iso_utc(utc_now())
try:
review = await judge_conversation(session_obj, samples, chat_client=client)
review.update(status="completed", model_config_id=runtime.id, reviewed_at=reviewed_at)
except Exception as exc:
_logger.warning("探索会话 %s judge 复核失败: %s", exploration_session_id, exc)
review = {
"status": "failed",
"model_config_id": runtime.id,
"reviewed_at": reviewed_at,
"error": str(exc)[:SAMPLE_TEXT_LIMIT],
}
session_obj.judge_review = review
repo.update(session_obj)
finally:
session.close()
_BACKGROUND_TASKS: set[asyncio.Task] = set()
def start_judge_review(exploration_session_id: str) -> asyncio.Task:
"""以后台任务启动抽样复核fire-and-forget结果经会话 judge_review 观测)。"""
task = asyncio.create_task(execute_judge_review(exploration_session_id))
# 强引用防止未持有引用的任务被 GCasyncio 已知坑)
_BACKGROUND_TASKS.add(task)
task.add_done_callback(_BACKGROUND_TASKS.discard)
return task

View File

@ -0,0 +1,29 @@
"""Exploration session settlement on campaign finalize (活动终态结算).
活动窗口结束或被取消进入终态时 running 的探索会话转 expired
不留悬挂会话结算后消息端点按既有状态机拒收409
"""
from sqlmodel import Session
from agenteval.exploration.models import ExplorationSessionStatus
from agenteval.storage.db import utc_now
from agenteval.storage.repository import ExplorationSessionRepository
def settle_campaign_sessions(campaign_id: str, session: Session) -> int:
"""Expire every still-running exploration session of a finalized campaign.
Returns the number of sessions expired. Completed/failed sessions keep
their evidence untouched.
"""
repo = ExplorationSessionRepository(session)
expired = 0
for session_obj in repo.list_by_campaign(campaign_id):
if session_obj.status != ExplorationSessionStatus.RUNNING:
continue
session_obj.status = ExplorationSessionStatus.EXPIRED
session_obj.closed_at = utc_now()
repo.update(session_obj)
expired += 1
return expired

View File

@ -0,0 +1,84 @@
"""Exploration findings aggregation (探索发现聚合).
把探索会话的体验记录聚合成探索摘要会话数目标达成率问题清单
blockers / misled 按出现次数降序judge 复核结论已就位才纳入
findings 全量收各档发现poor 档排前这是体验记录这条第一手证据线
进入报告 / 分析 / 导出三个出口前的唯一聚合口径只含统计与问题清单
不含全量对话
"""
from collections import Counter
from typing import Any, Optional
from sqlmodel import Session
from agenteval.exploration.models import ExplorationSession
_RATING_SEVERITY = {"poor": 0, "acceptable": 1, "good": 2}
def _issue_list(counter: Counter) -> list[dict[str, Any]]:
return [{"issue": issue, "count": count} for issue, count in counter.most_common()]
def summarize_exploration(sessions: list[ExplorationSession]) -> Optional[dict[str, Any]]:
"""聚合一组探索会话为探索摘要;无会话返回 None缺则无痕"""
if not sessions:
return None
with_experience = [s for s in sessions if s.experience is not None]
blockers: Counter = Counter()
misled: Counter = Counter()
achieved = 0
for session_obj in with_experience:
experience = session_obj.experience
if experience.get("goal_achieved"):
achieved += 1
blockers.update(experience.get("blockers") or [])
misled.update(experience.get("misled") or [])
judge = _summarize_judge_reviews(sessions)
return {
"session_count": len(sessions),
"sessions_with_experience": len(with_experience),
"goal_achieved_count": achieved,
"goal_achievement_rate": round(achieved / len(with_experience), 4) if with_experience else None,
"issues": _issue_list(blockers),
"misled": _issue_list(misled),
"judge_review": judge,
}
def summarize_campaign_exploration(db_session: Session, campaign_id: str) -> Optional[dict[str, Any]]:
"""取数 + 聚合一步完成:报告 / 分析 / 导出三个出口共用的探索摘要取法。"""
from agenteval.storage.repository import ExplorationSessionRepository
return summarize_exploration(ExplorationSessionRepository(db_session).list_by_campaign(campaign_id))
def _summarize_judge_reviews(sessions: list[ExplorationSession]) -> Optional[dict[str, Any]]:
"""只纳入复核完成的会话findings 全量收各档发现poor 档排前),
summaries 收复核总体结论"""
findings: list[dict[str, Any]] = []
summaries: list[str] = []
reviewed = 0
for session_obj in sessions:
review = session_obj.judge_review
if not review or review.get("status") != "completed":
continue
reviewed += 1
if review.get("summary"):
summaries.append(str(review["summary"]))
for item in review.get("dimensions") or []:
findings.append(
{
"dimension": item.get("dimension"),
"rating": item.get("rating"),
"comment": item.get("comment") or "",
}
)
if reviewed == 0:
return None
findings.sort(key=lambda f: _RATING_SEVERITY.get(f["rating"], len(_RATING_SEVERITY)))
return {"reviewed_sessions": reviewed, "findings": findings, "summaries": summaries}

View File

@ -26,6 +26,8 @@ from agenteval.evaluation.report import (
summarize_campaign_progress,
)
from agenteval.evaluation.report_render import render_campaign_markdown
from agenteval.exploration.settlement import settle_campaign_sessions
from agenteval.exploration.summary import summarize_campaign_exploration
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, ExplorationBudgetConfig, ExplorationSeeds
from agenteval.storage.db import iso_utc, utc_now
from agenteval.storage.model_config_repository import ModelConfigRepository
@ -120,6 +122,7 @@ async def cancel_campaign(campaign_id: str, session: Session = Depends(get_db))
campaign.completed_at = utc_now()
repo.update(campaign)
request_cancel(campaign_id)
settle_campaign_sessions(campaign_id, session)
return campaign.model_dump()
@ -130,7 +133,11 @@ async def get_campaign_report(campaign_id: str, session: Session = Depends(get_d
raise HTTPException(status_code=404, detail="campaign not found")
runs = RunRepository(session).list_by_campaign(campaign_id)
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
return generate_campaign_report(campaign, runs, scenario_names=scenario_names)
report = generate_campaign_report(campaign, runs, scenario_names=scenario_names)
exploration = summarize_campaign_exploration(session, campaign_id)
if exploration is not None:
report["exploration"] = exploration
return report
@router.get("/{campaign_id}/report/markdown")
@ -174,6 +181,7 @@ async def get_campaign_report_markdown(campaign_id: str, session: Session = Depe
generate_campaign_report(campaign, runs, scenario_names=scenario_names),
analysis=analysis,
comparison=comparison,
exploration=summarize_campaign_exploration(session, campaign_id),
target_name=target_name,
scenario_names=scenario_names,
)

View File

@ -10,6 +10,7 @@ with 409 plus a readable reason, so the rejection itself is feedback to the
resident agent.
"""
import json
from datetime import datetime
from typing import Any
@ -20,6 +21,7 @@ from sqlmodel import Session
from agenteval.channels.factory import ChannelFactory
from agenteval.config import get_settings
from agenteval.evaluation.report import generate_campaign_report
from agenteval.exploration.judge import start_judge_review
from agenteval.exploration.models import (
ExplorationBudget,
ExplorationMessage,
@ -60,6 +62,21 @@ class CloseSessionRequest(BaseModel):
experience: dict[str, Any]
def _coerce_reply_text(content: Any) -> str:
"""Flatten a reply payload to text; tutu returns msgBody as a parsed object,
and str(dict) would leak a Python repr into the view."""
if isinstance(content, str):
return content
if isinstance(content, dict):
for key in ("content", "text", "message"):
value = content.get(key)
if isinstance(value, str) and value:
return value
if content is None:
return ""
return json.dumps(content, ensure_ascii=False)
def _check_creation_guardrails(
campaign: Campaign,
triggered_by: ExplorationTrigger,
@ -200,6 +217,28 @@ async def create_session(
return repo.create(session_obj).model_dump(mode="json")
@router.get("/campaigns/{campaign_id}/sessions")
async def list_campaign_sessions(
campaign_id: str,
session: Session = Depends(get_db),
) -> dict:
if not CampaignRepository(session).get(campaign_id):
raise HTTPException(status_code=404, detail="campaign not found")
sessions = ExplorationSessionRepository(session).list_by_campaign(campaign_id)
return {"sessions": [s.model_dump(mode="json") for s in sessions]}
@router.get("/sessions/{session_id}/messages")
async def list_session_messages(
session_id: str,
session: Session = Depends(get_db),
) -> dict:
if not ExplorationSessionRepository(session).get(session_id):
raise HTTPException(status_code=404, detail="exploration session not found")
messages = ExplorationMessageRepository(session).list_by_session(session_id)
return {"messages": [m.model_dump(mode="json") for m in messages]}
@router.post("/sessions/{session_id}/messages")
async def send_session_message(
session_id: str,
@ -260,7 +299,7 @@ async def send_session_message(
received_at = utc_now()
latency_ms = int((received_at - sent_at).total_seconds() * 1000)
reply_text = str(reply.content)
reply_text = _coerce_reply_text(reply.content)
message_repo.save_message(
ExplorationMessage(
session_id=session_obj.id,
@ -290,4 +329,6 @@ async def close_session(
session_obj.experience = normalize_experience(request.experience)
session_obj.status = ExplorationSessionStatus.COMPLETED
session_obj.closed_at = utc_now()
return repo.update(session_obj).model_dump(mode="json")
updated = repo.update(session_obj)
start_judge_review(session_obj.id)
return updated.model_dump(mode="json")

View File

@ -32,7 +32,17 @@
部署脚本(`scripts/deploy-t480.sh`)会自动把它同步到 OpenClaw 工作区:
`data/openclaw/workspace/skills/agenteval-run/SKILL.md`
### 方式二Python Skill 封装
### 方式二agenteval-patrol 巡检技能探索式评测v0.9
`skills/agenteval-patrol/SKILL.md` 指导 AI 助手作为常驻巡检代理完成探索闭环:
调巡检 API`GET /api/exploration/patrol`)→ 研判新结果与预算余量 → 决定是否派发
探索会话 → 以虚拟用户身份驱动会话对话 → 提交体验记录关闭会话。说明书内含收到
409 时的收敛行为与衍生变体的种子回溯(`seed_ref`)要求。
部署脚本会把 `backend/plugins/openclaw/skills/` 下的每个技能同步到
`data/openclaw/workspace/skills/<skill-name>/SKILL.md`
### 方式三Python Skill 封装
参考 `agenteval_skill.py`httpx 调 API在 OpenClaw 中注册技能时传入配置:
@ -60,6 +70,21 @@ config:
report_format: "json"
```
### 常驻巡检作业(探索式评测)
巡检代理需注册为全局常驻作业,默认节拍 1 小时;调度状态由 OpenClaw 自身持久化,
重启后不丢:
```yaml
skill: agenteval-patrol
schedule: "0 * * * *" # 每小时整点巡检一次
config:
api_base_url: "http://agenteval:8000"
```
巡检 API 自带水位(`last_patrolled_at`),每个节拍只报增量结果,重复触发不会
重复上报;预算与间隔护栏在服务端强制执行,代理收到 409 即收敛。
## API 速查
| 操作 | 请求 |
@ -67,6 +92,10 @@ config:
| 启动评测 | `POST /api/runs` body: `{"target_id", "scenario_id", "triggered_by": "ai_assistant"}` |
| 查询状态 | `GET /api/runs/{run_id}`(轮询至 completed/failed |
| 获取报告 | `GET /api/reports/{run_id}` |
| 巡检探索 | `GET /api/exploration/patrol`(无参数,报增量新结果与预算余量) |
| 派发探索会话 | `POST /api/exploration/sessions` body: `{"campaign_id", "persona", "goal", "seed_ref?", "triggered_by"}` |
| 驱动会话对话 | `POST /api/exploration/sessions/{session_id}/messages` body: `{"content"}` |
| 提交体验记录 | `POST /api/exploration/sessions/{session_id}/close` body: `{"experience"}` |
如果平台设置了 `AGENTEVAL_API_KEY`,所有请求需带 `X-API-Key` 头。

View File

@ -0,0 +1,110 @@
---
name: agenteval-patrol
description: 常驻巡检:巡检 AgentEvalTool 评测活动,研判新结果与预算余量,自主派发并驱动探索式评测会话
---
你是平台的常驻巡检代理。本 skill 指导你按「巡检 → 研判 → 派发 → 驱动 → 提交」的闭环行动。
所有操作必须走 AgentEvalTool 标准 HTTP API禁止直接调 CLI 或操作数据库)。
平台可能启用了 API Key 鉴权。每次执行命令前先读取密钥(文件不存在则为空,不影响未启用鉴权的环境):
```bash
KEY=$(cat ~/.openclaw/agenteval-api-key 2>/dev/null)
```
以下所有 curl 命令都必须带 `-H "X-API-Key: $KEY"`
## 巡检(闭环第一步)
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/exploration/patrol | python3 -m json.tool
```
返回所有参与探索的进行中正式线活动。每个活动包含:
- `new_results`自上次巡检以来的新评测结果null 表示无新增)。有值时先消化它——通过率下滑的能力项是本次探索的优先方向。
- `budget`:探索预算台账。`remaining_sessions` 为剩余可派发会话数;`seconds_since_last_session` 与 `min_interval_seconds` 用于判断会话间隔是否已满足。
- 该接口自带水位:每次调用只报增量,无需自己记录上次结果。
## 研判与派发决策
对每个活动,按以下规则决定是否派发探索会话:
1. `remaining_sessions` 为 0 → 不派发(预算耗尽)。
2. `seconds_since_last_session` 非 null 且小于 `min_interval_seconds` → 不派发(间隔未到,等下个节拍)。
3. 其余情况可派发一个会话。persona 与 goal 必须取自活动的种子集:
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/campaigns/<campaign_id> | python3 -m json.tool
```
读取响应中的 `exploration_seeds.personas``exploration_seeds.goals`,各取其一组合。允许在种子上做衍生变体(如把「查账单」衍生为「查账单并开发票」),但必须在 `seed_ref` 里回溯来源。
## 派发探索会话
```bash
curl -s -X POST http://agenteval:8000/api/exploration/sessions \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"campaign_id": "<campaign_id>",
"persona": {"name": "<人设名>", "traits": ["<特质1>", "<特质2>"]},
"goal": "<探索目标>",
"seed_ref": {"persona_seed": "<所用 persona 种子原文>", "goal_seed": "<所用 goal 种子原文>", "derived": false},
"triggered_by": "auto"
}'
```
- 直接使用种子时 `derived` 为 false衍生变体时为 true并追加 `"derived_from": "<原种子>"``"variant": "<变异点说明>"`——留痕是探索数据可回溯的底线。
- 响应中的 `id` 即 session_id后续所有步骤都要用它。
## 驱动会话对话
以 persona 的身份向目标推进 goal每轮发送一条用户消息并同步收到回复
```bash
curl -s -X POST http://agenteval:8000/api/exploration/sessions/<session_id>/messages \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"content": "<本轮用户消息>"}'
```
- 根据 `reply` 决定下一轮:追问、纠正、换路径,直到目标达成或确认走不通。
- 单会话默认最多 12 轮,以 patrol 返回的 `budget.max_turns` 为准,接近上限时尽快收尾。
- 502 表示通道异常;连续两次 502 就结束会话,把异常写进体验记录备注。
## 提交体验记录(闭环收尾)
会话必须显式关闭,提交第一人称体验记录——这是探索证据链的源头,务必如实:
```bash
curl -s -X POST http://agenteval:8000/api/exploration/sessions/<session_id>/close \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"experience": {
"goal_achieved": true,
"blockers": ["<遇到的障碍逐条列出>"],
"misled": ["<被误导的经历回答把你带偏的地方>"],
"emotion": "positive",
"notes": "<整体体验简述>"
}
}'
```
- `emotion` 只能是 positive / neutral / confused / frustrated 之一,按真实体感选择。
- `blockers``misled` 写具体事实(如「让我重复提供订单号三次」),不写空泛评价。
## 收到 409 的收敛行为
409 是平台账本的明确拒绝,一律停止当前动作、不重试:
- 创建会话 409预算耗尽 / 间隔不足 / 活动非进行中)→ 本活动本轮放弃派发,记录原因,等下个巡检节拍。
- 发消息 409会话不在进行中 / 轮数超限)→ 立即走关闭流程提交已有体验记录;若关闭也返回 409说明会话已被平台收口如活动终态结算放弃并记录原因。
- 404活动或会话不存在→ 对象已消失,跳过。
## 汇报要求
每个巡检节拍结束后向用户汇报:巡检到的活动数、派发的会话数(含被 409 拦截的原因)、关闭的会话及其目标达成情况与关键问题。
请将 <campaign_id><session_id> 等占位符替换为实际值。

View File

@ -324,6 +324,69 @@ export interface ExplorationBudgetConfig {
min_interval_seconds?: number | null
}
export interface ExplorationIssueCount {
issue: string
count: number
}
export interface ExplorationJudgeReview {
reviewed_sessions: number
findings: { dimension: string; rating: string; comment: string }[]
summaries: string[]
}
export interface ExplorationSummary {
session_count: number
sessions_with_experience: number
goal_achieved_count: number
goal_achievement_rate: number | null
issues: ExplorationIssueCount[]
misled: ExplorationIssueCount[]
judge_review: ExplorationJudgeReview | null
}
export interface ExplorationExperience {
goal_achieved: boolean
blockers: string[]
misled: string[]
emotion: string
notes: string
}
export interface ExplorationSession {
id: string
campaign_id: string
target_id: string
persona: Record<string, unknown>
goal: string
seed_ref: Record<string, unknown> | null
status: string
triggered_by: string
experience: ExplorationExperience | null
judge_review: Record<string, unknown> | null
turn_count: number
error: string | null
created_at: string | null
closed_at: string | null
}
export interface ExplorationMessage {
id: string
session_id: string
round_index: number
role: string
content: string
latency_ms: number | null
created_at: string | null
}
export const explorationApi = {
listSessions: (campaignId: string) =>
api.get<{ sessions: ExplorationSession[] }>(`/exploration/campaigns/${campaignId}/sessions`),
listMessages: (sessionId: string) =>
api.get<{ messages: ExplorationMessage[] }>(`/exploration/sessions/${sessionId}/messages`),
}
export interface CampaignProgress {
current_offset_seconds: number
spawned_runs: number
@ -394,6 +457,7 @@ export interface CampaignReport {
}
time_trend: CampaignTrendBucket[]
capability_summary: CampaignCapability[]
exploration?: ExplorationSummary | null
}
export interface CampaignTimelineEntry {

View File

@ -7,9 +7,11 @@ interface ChatBubbleProps {
content: ReactNode
meta?: ReactNode
pending?: boolean
/** 镜像布局:探索下钻里虚拟用户居左、被评对象居右(与常规聊天相反) */
mirrored?: boolean
}
export default function ChatBubble({ role, content, meta, pending }: ChatBubbleProps) {
export default function ChatBubble({ role, content, meta, pending, mirrored }: ChatBubbleProps) {
const isUser = role === 'user'
const isAgent = role === 'agent'
const isSystem = role === 'system'
@ -17,8 +19,9 @@ export default function ChatBubble({ role, content, meta, pending }: ChatBubbleP
const bg = isUser ? colors.chatUser : isAgent ? colors.chatAgent : colors.chatSystemError
const border = isSystem ? '#ffccc7' : 'transparent'
const textColor = isSystem ? '#cf1322' : colors.text
const align = isUser ? 'flex-end' : 'flex-start'
const radius = isUser ? '10px 10px 2px 10px' : '10px 10px 10px 2px'
const alignRight = isSystem ? false : isUser ? !mirrored : !!mirrored
const align = alignRight ? 'flex-end' : 'flex-start'
const radius = alignRight ? '10px 10px 2px 10px' : '10px 10px 10px 2px'
const icon = isUser
? <UserOutlined style={{ color: colors.primary }} />
@ -29,7 +32,7 @@ export default function ChatBubble({ role, content, meta, pending }: ChatBubbleP
return (
<div style={{ display: 'flex', justifyContent: align, marginBottom: 6 }}>
<div style={{ maxWidth: '82%', display: 'flex', flexDirection: 'column', alignItems: isUser ? 'flex-end' : 'flex-start', gap: 2 }}>
<div style={{ maxWidth: '82%', display: 'flex', flexDirection: 'column', alignItems: alignRight ? 'flex-end' : 'flex-start', gap: 2 }}>
<div style={{ display: 'flex', gap: 4, alignItems: 'center', fontSize: 11, color: colors.textMuted }}>
{!isUser && icon}
<span>{label}</span>

View File

@ -0,0 +1,227 @@
import { useEffect, useState } from 'react'
import { Collapse, Empty, Space, Spin, Tag } from 'antd'
import ChatBubble from './ChatBubble'
import {
explorationApi,
type ExplorationMessage,
type ExplorationSession,
type ExplorationSummary,
} from '../api'
import { colors } from '../tokens'
import { shortDateTime } from '../utils/date'
const SESSION_STATUS: Record<string, { label: string; color: string }> = {
running: { label: '进行中', color: 'processing' },
completed: { label: '已完成', color: 'success' },
failed: { label: '失败', color: 'error' },
expired: { label: '已过期', color: 'default' },
}
const EMOTION_LABELS: Record<string, string> = {
positive: '满意',
neutral: '平静',
confused: '困惑',
frustrated: '沮丧',
}
const DIMENSION_LABELS: Record<string, string> = {
attitude: '服务态度',
professionalism: '专业度',
hallucination: '幻觉',
}
const RATING_TAG: Record<string, { label: string; color: string }> = {
poor: { label: '差', color: 'red' },
acceptable: { label: '一般', color: 'orange' },
good: { label: '优', color: 'green' },
}
function pct(rate: number | null): string {
return rate == null ? '—' : `${(rate * 100).toFixed(1)}%`
}
function IssueList({ label, items, misled }: {
label: string
items: { issue: string; count: number }[]
misled?: boolean
}) {
if (items.length === 0) return null
return (
<div style={{ marginBottom: 10 }}>
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 6 }}>{label}</div>
<Space size={8} wrap>
{items.map((item, i) => (
<Tag key={i} color={misled ? 'orange' : 'red'}>
{misled ? `(被误导)${item.issue}` : item.issue} ×{item.count}
</Tag>
))}
</Space>
</div>
)
}
function SessionDetail({ session, messages }: {
session: ExplorationSession
messages: ExplorationMessage[] | null
}) {
const experience = session.experience
return (
<div>
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 10 }}>
{session.goal}
{session.created_at && <> · {shortDateTime(session.created_at)}</>}
{session.closed_at && <> · {shortDateTime(session.closed_at)}</>}
</div>
{!messages && <Spin size="small" />}
{messages && messages.length === 0 && (
<Empty description="暂无对话记录" image={Empty.PRESENTED_IMAGE_SIMPLE} />
)}
{messages && messages.map((m) => (
<ChatBubble
key={m.id}
role={m.role === 'user' ? 'user' : 'agent'}
mirrored
content={m.content}
meta={m.role !== 'user' && m.latency_ms != null ? (
<span style={{ color: colors.textMuted, fontSize: 11 }}>{m.latency_ms}ms</span>
) : undefined}
/>
))}
{experience && (
<div style={{
marginTop: 8, border: `1px solid ${colors.border}`, borderRadius: 8,
background: colors.bgSubtle, padding: '10px 12px', fontSize: 13, lineHeight: 1.8,
}}>
<div style={{ fontWeight: 500, marginBottom: 4 }}></div>
<Space size={8} wrap>
{experience.goal_achieved
? <Tag color="success"></Tag>
: <Tag color="error"></Tag>}
<Tag>{EMOTION_LABELS[experience.emotion] ?? experience.emotion}</Tag>
</Space>
{experience.blockers.length > 0 && (
<div style={{ color: colors.textSecondary }}>{experience.blockers.join('')}</div>
)}
{experience.misled.length > 0 && (
<div style={{ color: colors.warning }}>{experience.misled.join('')}</div>
)}
{experience.notes && (
<div style={{ color: colors.textSecondary }}>{experience.notes}</div>
)}
</div>
)}
{session.error && (
<div style={{ marginTop: 8, fontSize: 12, color: colors.textMuted }}>{session.error}</div>
)}
</div>
)
}
export default function ExplorationSection({ campaignId, summary }: {
campaignId: string
summary: ExplorationSummary
}) {
const [sessions, setSessions] = useState<ExplorationSession[] | null>(null)
const [loading, setLoading] = useState(false)
const [messageCache, setMessageCache] = useState<Record<string, ExplorationMessage[]>>({})
useEffect(() => {
let cancelled = false
setLoading(true)
explorationApi.listSessions(campaignId)
.then((res) => { if (!cancelled) setSessions(res.data.sessions) })
.catch(() => undefined) // 拦截器已弹错;避免未处理 rejection
.finally(() => { if (!cancelled) setLoading(false) })
return () => { cancelled = true }
}, [campaignId])
const onExpand = (keys: string | string[]) => {
for (const key of Array.isArray(keys) ? keys : [keys]) {
if (key in messageCache) continue
void explorationApi.listMessages(key)
.then((res) => { setMessageCache((prev) => ({ ...prev, [key]: res.data.messages })) })
.catch(() => undefined) // 拦截器已弹错;收起再展开可重试
}
}
const judge = summary.judge_review
return (
<div style={{
border: `1px solid ${colors.border}`, borderRadius: 8,
padding: '12px 14px', marginBottom: 16,
}}>
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 10 }}>
{summary.session_count} · {summary.sessions_with_experience}
{' · '} {pct(summary.goal_achievement_rate)}{summary.goal_achieved_count}/{summary.sessions_with_experience}
</div>
<IssueList label="问题清单(体验记录)" items={summary.issues} />
<IssueList label="被误导清单(体验记录)" items={summary.misled} misled />
{summary.issues.length === 0 && summary.misled.length === 0 && (
<div style={{ fontSize: 12, color: colors.textMuted, marginBottom: 10 }}></div>
)}
{judge && (
<div style={{ marginBottom: 10 }}>
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 6 }}>
judge {judge.reviewed_sessions}
</div>
{judge.findings.length === 0
? <div style={{ fontSize: 13, color: colors.textSecondary }}></div>
: (
<Space direction="vertical" size={4} style={{ width: '100%' }}>
{judge.findings.map((f, i) => (
<div key={i} style={{ fontSize: 13 }}>
<Tag color={RATING_TAG[f.rating]?.color ?? 'default'} style={{ margin: 0 }}>
{DIMENSION_LABELS[f.dimension] ?? f.dimension}
{' · '}
{RATING_TAG[f.rating]?.label ?? f.rating}
</Tag>
{' '}{f.comment}
</div>
))}
</Space>
)}
{judge.summaries.map((s, i) => (
<div key={i} style={{ fontSize: 13, color: colors.textSecondary, marginTop: 4 }}>
{s}
</div>
))}
</div>
)}
{loading && <Spin size="small" />}
{!loading && sessions && sessions.length === 0 && (
<Empty description="暂无探索会话" image={Empty.PRESENTED_IMAGE_SIMPLE} />
)}
{sessions && sessions.length > 0 && (
<Collapse
size="small"
onChange={onExpand}
items={sessions.map((s) => {
const meta = SESSION_STATUS[s.status] ?? SESSION_STATUS.running
const personaName = typeof s.persona?.name === 'string' ? s.persona.name : s.id.slice(0, 8)
return {
key: s.id,
label: (
<Space size={8} wrap>
<span style={{ fontWeight: 500 }}>{personaName}</span>
<Tag color={meta.color}>{meta.label}</Tag>
<Tag>{s.turn_count} </Tag>
{s.experience?.goal_achieved != null && (
s.experience.goal_achieved
? <Tag color="success"></Tag>
: <Tag color="error"></Tag>
)}
</Space>
),
children: <SessionDetail session={s} messages={messageCache[s.id] ?? null} />,
}
})}
/>
)}
</div>
)
}

View File

@ -8,7 +8,7 @@ import {
PlusOutlined, ReloadOutlined, StopOutlined, BarChartOutlined,
FileMarkdownOutlined, MinusCircleOutlined, QuestionCircleOutlined,
RocketOutlined, CheckCircleOutlined, SafetyOutlined, ClockCircleOutlined,
RobotOutlined, SwapOutlined,
RobotOutlined, SwapOutlined, CompassOutlined,
} from '@ant-design/icons'
import { Line, Bar } from '@ant-design/charts'
import PageWrapper from '../components/PageWrapper'
@ -25,6 +25,7 @@ import { deriveTimeScale, acceleratedDuration, formatScale } from '../utils/camp
import WindowTimeline, { type TimelineMarker } from '../components/WindowTimeline'
import CampaignRunTimeline from '../components/CampaignRunTimeline'
import PeriodComparisonSection from '../components/PeriodComparisonSection'
import ExplorationSection from '../components/ExplorationSection'
import { useResource } from '../hooks/useResource'
import { usePolling } from '../hooks/usePolling'
import { useTabStore } from '../stores/tabStore'
@ -1055,6 +1056,17 @@ export default function CampaignsPage() {
onGenerate={generateComparison}
/>
{report.exploration && (
<>
<SectionTitle><CompassOutlined /> </SectionTitle>
<ExplorationSection
key={report.campaign_id}
campaignId={report.campaign_id}
summary={report.exploration}
/>
</>
)}
<SectionTitle></SectionTitle>
<div style={{
border: `1px solid ${colors.border}`, borderRadius: 8,

View File

@ -93,14 +93,16 @@ run rsync -az --delete \
./ "$HOST:$REMOTE_DIR/"
# ── OpenClaw skill sync ────────────────────────────────────────────────
# The agenteval-run skill lives in the openclaw workspace volume (data/),
# which rsync excludes. Sync the versioned copy explicitly so the AI
# assistant always follows the standard HTTP API flow.
SKILL_SRC="backend/plugins/openclaw/skills/agenteval-run/SKILL.md"
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/agenteval-run/SKILL.md"
log "sync OpenClaw skill → $SKILL_DST"
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
# OpenClaw skills live in the openclaw workspace volume (data/), which
# rsync excludes. Sync every versioned skill explicitly so the AI
# assistant always follows the standard HTTP API flows.
for SKILL_SRC in backend/plugins/openclaw/skills/*/SKILL.md; do
SKILL_NAME=$(basename "$(dirname "$SKILL_SRC")")
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/$SKILL_NAME/SKILL.md"
log "sync OpenClaw skill → $SKILL_DST"
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
done
# Provision the API key file the skill reads (~/.openclaw/agenteval-api-key in
# the openclaw container = data/openclaw/agenteval-api-key on the host). Sourced

View File

@ -302,6 +302,61 @@ async def test_campaign_report_markdown_export(client, seeded_db):
assert "## 能力汇总" in resp.text
# ── exploration findings in report/export (v0.9 ticket 05) ───────────────────
def _seed_exploration_session(seeded_db, campaign_id: str) -> None:
from agenteval.exploration.models import ExplorationSession
from agenteval.storage.repository import ExplorationSessionRepository
repo = ExplorationSessionRepository(seeded_db)
session_obj = repo.create(
ExplorationSession(campaign_id=campaign_id, target_id="t-1", persona={"name": "x"}, goal="缴费")
)
session_obj.experience = {
"goal_achieved": True,
"blockers": ["缴费入口难找"],
"misled": [],
"emotion": "neutral",
"notes": "",
}
repo.update(session_obj)
async def test_campaign_report_includes_exploration_findings(client, seeded_db):
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
_seed_exploration_session(seeded_db, campaign_id)
report = (await client.get(f"/api/campaigns/{campaign_id}/report")).json()
exploration = report["exploration"]
assert exploration["session_count"] == 1
assert exploration["goal_achievement_rate"] == 1.0
assert exploration["issues"] == [{"issue": "缴费入口难找", "count": 1}]
async def test_campaign_report_without_exploration_omits_key(client, seeded_db):
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
report = (await client.get(f"/api/campaigns/{campaign_id}/report")).json()
assert "exploration" not in report
async def test_markdown_export_appends_exploration_appendix(client, seeded_db):
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
_seed_exploration_session(seeded_db, campaign_id)
resp = await client.get(f"/api/campaigns/{campaign_id}/report/markdown")
assert resp.status_code == 200
assert "## 探索发现" in resp.text
assert "缴费入口难找 ×1" in resp.text
async def test_markdown_export_without_exploration_leaves_no_trace(client, seeded_db):
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
resp = await client.get(f"/api/campaigns/{campaign_id}/report/markdown")
assert resp.status_code == 200
assert "探索发现" not in resp.text
# ── campaign timeline endpoint (ticket 06) ───────────────────────────────────
async def test_campaign_timeline_structure(client, seeded_db):

View File

@ -54,6 +54,11 @@ def seeded_db(db_session, monkeypatch):
app.dependency_overrides[get_db] = _test_get_db
# 隔离后台 judge 复核任务API 测试不真正派发后台任务
from agenteval.web.routers import exploration as exploration_router
monkeypatch.setattr(exploration_router, "start_judge_review", lambda session_id: None)
target = EvalTarget(
id="t-1",
name="mock-target",
@ -170,6 +175,37 @@ async def test_full_lifecycle_create_message_close(seeded_db, mock_channel, clie
assert closed["closed_at"] is not None
async def test_dict_reply_content_is_flattened_to_text(seeded_db, monkeypatch, client):
"""通道回复 content 为对象(如 tutu msgBody时应提取文本而非存 str(dict)。"""
from agenteval.channels.base import Reply
from tests.unit.mock_channel import MockChannel
class _DictReplyChannel(MockChannel):
async def poll_reply(self, question_msg_id, timeout=30.0, poll_interval=1.0):
return Reply(
question_msg_id=question_msg_id,
content={"content": "您好,我是客服"},
raw_message={},
)
_stub_channel_factory(monkeypatch, _DictReplyChannel(reply_text="unused"))
resp = await _create_session(client)
assert resp.status_code == 200, resp.text
session_id = resp.json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "你好"},
)
assert resp.status_code == 200, resp.text
assert resp.json()["reply"] == "您好,我是客服"
messages_resp = await client.get(f"/api/exploration/sessions/{session_id}/messages")
assistant = [m for m in messages_resp.json()["messages"] if m["role"] == "assistant"]
assert assistant[0]["content"] == "您好,我是客服"
async def test_create_requires_existing_running_campaign(seeded_db, client):
resp = await _create_session(client, campaign_id="nope")
assert resp.status_code == 404
@ -330,6 +366,80 @@ async def test_close_twice_rejected(seeded_db, mock_channel, client):
assert resp.status_code == 409
async def test_close_triggers_judge_review(seeded_db, mock_channel, client, monkeypatch):
from agenteval.web.routers import exploration as exploration_router
started: list[str] = []
monkeypatch.setattr(exploration_router, "start_judge_review", started.append)
session_id = (await _create_session(client)).json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close", json={"experience": {"goal_achieved": True}}
)
assert resp.status_code == 200, resp.text
assert started == [session_id]
# ---------------------------------------------------------------- read endpoints (ticket 06)
async def test_list_campaign_sessions_returns_lifecycle_fields(seeded_db, mock_channel, client):
resp = await _create_session(client)
assert resp.status_code == 200, resp.text
session_id = resp.json()["id"]
await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "查账单"})
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close",
json={"experience": {"goal_achieved": True, "blockers": [], "misled": ["跳转误导"], "notes": "绕了三圈"}},
)
assert resp.status_code == 200, resp.text
resp = await client.get("/api/exploration/campaigns/c-1/sessions")
assert resp.status_code == 200, resp.text
sessions = resp.json()["sessions"]
assert len(sessions) == 1
entry = sessions[0]
assert entry["id"] == session_id
assert entry["status"] == "completed"
assert entry["turn_count"] == 1
assert entry["goal"] == "查询本月账单并完成缴费"
assert entry["persona"]["name"] == "急性子用户"
assert entry["experience"]["goal_achieved"] is True
assert entry["experience"]["misled"] == ["跳转误导"]
assert entry["closed_at"] is not None
async def test_list_sessions_empty_campaign(seeded_db, client):
resp = await client.get("/api/exploration/campaigns/c-1/sessions")
assert resp.status_code == 200, resp.text
assert resp.json() == {"sessions": []}
async def test_list_session_messages_returns_conversation(seeded_db, mock_channel, client):
session_id = (await _create_session(client)).json()["id"]
await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "我要查账单"})
resp = await client.get(f"/api/exploration/sessions/{session_id}/messages")
assert resp.status_code == 200, resp.text
messages = resp.json()["messages"]
assert len(messages) == 2
user_msg, agent_msg = messages
assert user_msg["role"] == "user"
assert user_msg["content"] == "我要查账单"
assert user_msg["round_index"] == 1
assert agent_msg["role"] == "assistant"
assert agent_msg["content"] == "您好,请问有什么可以帮您?"
assert agent_msg["round_index"] == 1
assert isinstance(agent_msg["latency_ms"], int)
assert user_msg["created_at"] is not None
async def test_list_messages_unknown_session_returns_404(seeded_db, client):
resp = await client.get("/api/exploration/sessions/nope/messages")
assert resp.status_code == 404
# ---------------------------------------------------------------- migration

View File

@ -0,0 +1,189 @@
"""Integration tests for exploration settlement on campaign finalize (v0.9 票据 07).
Covers both terminal transitions: window completion (durable loop COMPLETE)
and mid-flight cancellation. Running sessions must turn expired and reject
further messages (409), leaving no dangling sessions.
"""
import asyncio
import pytest
from agenteval.evaluation import campaign_runner as runner_module
from agenteval.evaluation.campaign_runner import run_campaign_loop
from agenteval.exploration.models import ExplorationSessionStatus
from agenteval.models import (
Campaign,
CampaignPlanEntry,
Case,
CaseType,
ChannelType,
EvalTarget,
PlatformType,
Scenario,
TargetStatus,
)
from agenteval.storage.db import utc_now
from agenteval.storage.repository import (
CampaignRepository,
ExplorationSessionRepository,
ScenarioRepository,
TargetRepository,
)
from agenteval.web.app import app
from httpx import ASGITransport, AsyncClient
def _session_payload(campaign_id: str) -> dict:
return {
"campaign_id": campaign_id,
"persona": {"name": "巡检用户", "traits": ["耐心"]},
"goal": "查询账单",
"triggered_by": "manual",
}
@pytest.fixture()
def seeded_db(db_session, monkeypatch):
"""Point every get_session consumer at the test session and seed the
target + scenario needed by both the loop path and the cancel path."""
from agenteval.channels import factory as factory_module
from agenteval.evaluation import engine as engine_module
from agenteval.storage import db as db_module
from agenteval.storage import repository as repo_module
from agenteval.web import app as app_module
from agenteval.web.routers import exploration as exploration_router
monkeypatch.setattr(app_module, "init_db", lambda: None)
def _test_get_session():
return db_session
monkeypatch.setattr(db_module, "get_session", _test_get_session)
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
monkeypatch.setattr(engine_module, "get_session", _test_get_session)
monkeypatch.setattr(runner_module, "get_session", _test_get_session)
monkeypatch.setattr(runner_module, "_auto_start_analysis", lambda campaign, session: None)
from tests.unit.mock_channel import MockChannel
channel = MockChannel(reply_delay=0.0)
monkeypatch.setattr(factory_module.ChannelFactory, "create", lambda target: channel)
monkeypatch.setattr(exploration_router, "start_judge_review", lambda session_id: None)
from agenteval.web.deps import get_db
def _test_get_db():
try:
yield db_session
finally:
pass
app.dependency_overrides[get_db] = _test_get_db
target = EvalTarget(
id="t-1",
name="mock-target",
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API,
channel_config={"base_url": "http://mock", "token": "x"},
status=TargetStatus.ACTIVE,
)
TargetRepository(db_session).create(target)
scenario = Scenario(
id="s-1",
name="mock-scenario",
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
)
ScenarioRepository(db_session).create(scenario)
yield db_session
app.dependency_overrides.clear()
def _make_campaign(db_session, campaign_id: str, *, time_scale: float) -> Campaign:
return CampaignRepository(db_session).create(
Campaign(
id=campaign_id,
name=f"campaign-{campaign_id}",
target_id="t-1",
window_seconds=3600,
time_scale=time_scale,
plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=1)],
status="running",
started_at=utc_now(),
)
)
async def _create_running_session(db_session, campaign_id: str) -> str:
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
resp = await c.post("/api/exploration/sessions", json=_session_payload(campaign_id))
assert resp.status_code == 200, resp.text
session_id = resp.json()["id"]
assert ExplorationSessionRepository(db_session).get(session_id).status == (ExplorationSessionStatus.RUNNING)
return session_id
async def test_window_completion_expires_running_sessions(seeded_db):
campaign = _make_campaign(seeded_db, "c-done", time_scale=3600.0)
session_id = await _create_running_session(seeded_db, "c-done")
# time_scale 3600 → 1 real second covers the whole window; the loop
# completes on its first tick and settles exploration sessions.
cancel = asyncio.Event()
await asyncio.wait_for(run_campaign_loop(campaign.id, cancel, tick_seconds=0.05), timeout=5)
assert CampaignRepository(seeded_db).get("c-done").status.value == "completed"
settled = ExplorationSessionRepository(seeded_db).get(session_id)
assert settled.status == ExplorationSessionStatus.EXPIRED
assert settled.closed_at is not None
async def test_cancel_expires_running_sessions_and_rejects_messages(seeded_db):
_make_campaign(seeded_db, "c-1", time_scale=1.0)
session_id = await _create_running_session(seeded_db, "c-1")
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
resp = await client.post("/api/campaigns/c-1/cancel")
assert resp.status_code == 200, resp.text
assert resp.json()["status"] == "cancelled"
settled = ExplorationSessionRepository(seeded_db).get(session_id)
assert settled.status == ExplorationSessionStatus.EXPIRED
assert settled.closed_at is not None
resp = await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "还在吗"})
assert resp.status_code == 409
async def test_completed_session_survives_settlement(seeded_db):
_make_campaign(seeded_db, "c-1", time_scale=1.0)
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
resp = await client.post("/api/exploration/sessions", json=_session_payload("c-1"))
session_id = resp.json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close",
json={"experience": {"goal_achieved": True}},
)
assert resp.status_code == 200
await client.post("/api/campaigns/c-1/cancel")
assert ExplorationSessionRepository(seeded_db).get(session_id).status == (ExplorationSessionStatus.COMPLETED)
async def test_production_line_manual_trigger_also_settles(seeded_db):
_make_campaign(seeded_db, "c-1", time_scale=1.0)
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
resp = await client.post(
"/api/exploration/sessions",
json={**_session_payload("c-1"), "triggered_by": "manual"},
)
assert resp.status_code == 200, resp.text
session_id = resp.json()["id"]
assert resp.json()["triggered_by"] == "manual"
# 正式线手动会话同样纳入终态结算,触发来源不影响收口。
await client.post("/api/campaigns/c-1/cancel")
assert ExplorationSessionRepository(seeded_db).get(session_id).status == (ExplorationSessionStatus.EXPIRED)

View File

@ -152,6 +152,45 @@ async def test_scenario_without_failure_samples_still_gets_narrative():
assert len(result["scenario_narratives"]) == 2
async def test_analysis_input_includes_exploration_summary():
"""v0.9 票据 05阶段二输入追加探索摘要问题清单 + 达成统计,非全量对话)。"""
exploration_summary = {
"session_count": 2,
"sessions_with_experience": 2,
"goal_achieved_count": 1,
"goal_achievement_rate": 0.5,
"issues": [{"issue": "缴费入口难找", "count": 2}],
"misled": [],
"judge_review": None,
}
client = FakeChatClient(STAGE1_A, STAGE1_B, STAGE2)
await analyze_campaign(
campaign=_campaign(),
report=_report(),
failure_samples={},
valid_run_ids=set(),
chat_client=client,
exploration_summary=exploration_summary,
)
stage2_prompt = json.dumps(client.calls[2], ensure_ascii=False)
assert "探索发现" in stage2_prompt
assert "缴费入口难找" in stage2_prompt
assert "goal_achievement_rate" in stage2_prompt or "0.5" in stage2_prompt
async def test_analysis_without_exploration_omits_section():
client = FakeChatClient(STAGE1_A, STAGE1_B, STAGE2)
await analyze_campaign(
campaign=_campaign(),
report=_report(),
failure_samples={},
valid_run_ids=set(),
chat_client=client,
)
stage2_prompt = json.dumps(client.calls[2], ensure_ascii=False)
assert "探索发现" not in stage2_prompt
# ── 分析模型解析 ─────────────────────────────────────────────────────────
def _seed_config(session, config_id: str, *, analysis_default: bool = False, enabled: bool = True) -> None:

View File

@ -0,0 +1,235 @@
"""Report aggregation logic & output format validation (报告聚合与格式校验).
补充 test_campaign_report.py数值口径 test_metrics.py单点聚合之外
- 聚合边界用户取消不计分母ADR-0004窗口外/窗口前运行钳制
SQLite naive 时间戳与 aware 等价
- 格式校验报告 dict 的键集合类型取值范围时间序列连续性
能力面排序与 ISO-UTC 时间戳格式generate_campaign_report 是纯函数
直接构造模型对象不落库
"""
from datetime import datetime, timedelta, timezone
from agenteval.evaluation.metrics import aggregate_runs
from agenteval.evaluation.report import generate_campaign_report
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, EvalRun, RunStatus
T0 = datetime(2026, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
def _campaign(*, window_seconds: int = 12, time_scale: float = 1.0, started_at=T0, completed_at=None) -> Campaign:
return Campaign(
id="camp-fmt",
name="格式校验活动",
target_id="t-1",
window_seconds=window_seconds,
time_scale=time_scale,
plan=[CampaignPlanEntry(scenario_id="s-a", offset_seconds=0, count=1)],
status=CampaignStatus.RUNNING,
started_at=started_at,
completed_at=completed_at,
)
def _run(
scenario_id: str = "s-a",
*,
status: RunStatus = RunStatus.COMPLETED,
offset_seconds: float = 0.0,
pass_rate: float | None = None,
latency: float | None = None,
error: dict | None = None,
started_at: datetime | None = None,
) -> EvalRun:
summary = None
if pass_rate is not None or latency is not None or error is not None:
summary = {}
if pass_rate is not None:
summary["pass_rate"] = pass_rate
if latency is not None:
summary["avg_latency_ms"] = latency
if error is not None:
summary["error"] = error
return EvalRun(
target_id="t-1",
scenario_id=scenario_id,
campaign_id="camp-fmt",
status=status,
started_at=started_at if started_at is not None else T0 + timedelta(seconds=offset_seconds),
summary=summary,
)
def _report(runs, **kwargs):
return generate_campaign_report(_campaign(), runs, **kwargs)
# ── 格式校验:顶层结构 ────────────────────────────────────────────────
def test_top_level_schema_keys_and_types():
report = _report([_run(pass_rate=1.0, latency=100)])
assert set(report) == {
"campaign_id",
"name",
"target_id",
"status",
"window_seconds",
"time_scale",
"started_at",
"completed_at",
"summary",
"time_trend",
"capability_summary",
}
assert report["campaign_id"] == "camp-fmt"
assert isinstance(report["window_seconds"], int) and report["window_seconds"] > 0
assert isinstance(report["time_scale"], float)
assert report["status"] == CampaignStatus.RUNNING.value
assert isinstance(report["time_trend"], list)
assert isinstance(report["capability_summary"], list)
def test_summary_schema_types_and_bounds():
report = _report(
[
_run(pass_rate=1.0, latency=100),
_run(scenario_id="s-b", status=RunStatus.FAILED),
]
)
summary = report["summary"]
assert set(summary) == {
"total_runs",
"completed_runs",
"overall_pass_rate",
"overall_availability",
"avg_latency_ms",
}
assert isinstance(summary["total_runs"], int) and summary["total_runs"] == 2
assert isinstance(summary["completed_runs"], int) and summary["completed_runs"] == 1
for key in ("overall_pass_rate", "overall_availability"):
value = summary[key]
assert value is None or (isinstance(value, float) and 0.0 <= value <= 1.0), key
assert summary["avg_latency_ms"] is None or summary["avg_latency_ms"] >= 0
def test_summary_matches_single_aggregation_seam():
"""summary 数值必须与单点聚合口径 aggregate_runs 完全一致ADR-0004"""
runs = [
_run(pass_rate=1.0, latency=100),
_run(pass_rate=0.5, latency=300),
_run(status=RunStatus.FAILED),
]
report = _report(runs)
agg = aggregate_runs(runs)
summary = report["summary"]
assert summary["overall_pass_rate"] == agg["pass_rate"]
assert summary["overall_availability"] == agg["availability"]
assert summary["avg_latency_ms"] == agg["avg_latency_ms"]
# ── 格式校验:时间趋势序列 ────────────────────────────────────────────
def test_time_trend_contiguous_buckets_with_custom_count():
report = generate_campaign_report(_campaign(window_seconds=12), [_run(pass_rate=1.0)], bucket_count=4)
trend = report["time_trend"]
assert len(trend) == 4
for idx, bucket in enumerate(trend):
assert bucket["bucket_index"] == idx
assert bucket["start_seconds"] == idx * 3.0 # window / bucket_count
assert bucket["end_seconds"] == (idx + 1) * 3.0
assert {"run_count", "pass_rate", "availability", "avg_latency_ms"} <= set(bucket)
# 相邻桶首尾相接
for prev, cur in zip(trend, trend[1:]):
assert prev["end_seconds"] == cur["start_seconds"]
def test_time_trend_bucket_types_and_bounds():
report = _report([_run(pass_rate=0.8, latency=120)])
for bucket in report["time_trend"]:
assert isinstance(bucket["run_count"], int) and bucket["run_count"] >= 0
if bucket["run_count"] == 0:
assert bucket["pass_rate"] is None and bucket["availability"] is None
else:
assert 0.0 <= bucket["pass_rate"] <= 1.0
assert 0.0 <= bucket["availability"] <= 1.0
# ── 格式校验:能力面汇总 ──────────────────────────────────────────────
def test_capability_summary_schema_and_sorted_by_run_count_desc():
runs = [
_run("s-a", pass_rate=1.0),
_run("s-a", pass_rate=0.5),
_run("s-a", pass_rate=0.0),
_run("s-b", pass_rate=1.0),
]
report = _report(runs, scenario_names={"s-a": "夜间问诊"})
caps = report["capability_summary"]
assert [c["scenario_id"] for c in caps] == ["s-a", "s-b"] # run_count 降序
for cap in caps:
assert {"scenario_id", "scenario_name", "run_count", "pass_rate", "availability", "avg_latency_ms"} <= set(cap)
assert cap["run_count"] >= 1
assert caps[0]["scenario_name"] == "夜间问诊"
assert caps[1]["scenario_name"] == "s-b"[:8] # 无映射时回落 id 前缀
# ── 格式校验:时间戳 ISO-UTC ──────────────────────────────────────────
def test_timestamps_serialize_as_iso_utc_or_none():
report = generate_campaign_report(_campaign(completed_at=T0 + timedelta(hours=1)), [_run()])
assert report["started_at"].endswith("Z")
assert report["completed_at"].endswith("Z")
assert "T" in report["started_at"]
unfinished = generate_campaign_report(_campaign(), [_run()])
assert unfinished["completed_at"] is None
def test_naive_campaign_timestamps_still_serialize_utc():
naive_start = datetime(2026, 1, 1, 0, 0, 0) # SQLite round-trip drops tzinfo
report = generate_campaign_report(_campaign(started_at=naive_start), [_run()])
assert report["started_at"].endswith("Z")
# ── 聚合边界:取消 / 钳制 / naive 时间 ────────────────────────────────
def test_cancelled_run_excluded_from_summary_but_counted():
runs = [
_run(pass_rate=1.0, latency=100),
_run(status=RunStatus.FAILED, error={"code": "cancelled_by_user", "message": "stop"}),
]
summary = _report(runs)["summary"]
assert summary["total_runs"] == 2 # 发生过的事仍然可见
assert summary["completed_runs"] == 1
assert summary["overall_pass_rate"] == 1.0 # 取消不进分母ADR-0004
assert summary["overall_availability"] == 1.0
def test_run_started_before_window_clamped_to_first_bucket():
report = _report([_run(pass_rate=1.0, started_at=T0 - timedelta(seconds=10))])
assert report["time_trend"][0]["run_count"] == 1
def test_run_started_after_window_clamped_to_last_bucket():
report = _report([_run(pass_rate=1.0, started_at=T0 + timedelta(seconds=100))])
assert report["time_trend"][-1]["run_count"] == 1
assert sum(b["run_count"] for b in report["time_trend"]) == 1
def test_naive_run_timestamps_bucket_same_as_aware():
aware = _run(pass_rate=1.0, started_at=T0 + timedelta(seconds=6))
naive = _run(pass_rate=1.0, started_at=datetime(2026, 1, 1, 0, 0, 6)) # SQLite 读回的 naive UTC
trend_aware = _report([aware])["time_trend"]
trend_naive = _report([naive])["time_trend"]
assert trend_aware == trend_naive
assert trend_naive[6]["run_count"] == 1 # window 12s / 12 buckets → offset 6 落第 6 桶
def test_unstarted_run_sits_at_offset_zero():
report = _report([_run(pass_rate=1.0, started_at=None)])
assert report["time_trend"][0]["run_count"] == 1

View File

@ -0,0 +1,239 @@
"""Judge sampling review for exploration sessions (v0.9 票据 04).
会话关闭后平台抽样对话3 judge 岗位模型独立复核质量维度结论
结构化落入会话的 judge_review复核是异步后台执行失败落错误不阻塞会话
状态无模型配置时静默跳过LLM 调用经可注入 ChatClient 接缝沿 v0.7
分析 seam测试用假客户端覆盖先例test_campaign_analysis.py
"""
import json
import pytest
from agenteval.exploration.judge import (
MAX_JUDGE_SAMPLES,
execute_judge_review,
normalize_judge_review,
sample_round_indexes,
)
from agenteval.exploration.models import (
ExplorationMessage,
ExplorationSession,
ExplorationSessionStatus,
)
from agenteval.models import Campaign, CampaignPlanEntry
from agenteval.storage.db import ModelConfigDB
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import (
CampaignRepository,
ExplorationMessageRepository,
ExplorationSessionRepository,
)
class FakeChatClient:
"""Queued-response fake for the judge LLM seam."""
def __init__(self, *responses):
self._responses = list(responses)
self.calls: list[list[dict]] = []
async def __call__(self, messages: list[dict]) -> str:
self.calls.append(messages)
if not self._responses:
raise AssertionError("unexpected extra LLM call")
item = self._responses.pop(0)
if isinstance(item, Exception):
raise item
return item
REVIEW_JSON = json.dumps(
{
"dimensions": [
{"dimension": "attitude", "rating": "good", "comment": "态度友好"},
{"dimension": "professionalism", "rating": "acceptable", "comment": "流程基本正确"},
{"dimension": "hallucination", "rating": "poor", "comment": "编造了不存在的政策"},
],
"summary": "服务态度好但存在幻觉",
},
ensure_ascii=False,
)
def _campaign() -> Campaign:
return Campaign(
id="camp-1",
name="24h 正式线",
target_id="t-1",
window_seconds=86400,
time_scale=1.0,
plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=1)],
)
def _seed_config(db_session, config_id: str = "mc-default") -> None:
ModelConfigRepository(db_session).create(
ModelConfigDB(
id=config_id,
name=f"cfg-{config_id}",
provider="openai_compatible",
capability="chat",
endpoint_url="https://models.example.com/v1/chat/completions",
model_name="m",
enabled=True,
is_analysis_default=True,
)
)
def _seed_session_with_messages(db_session, rounds: int = 5) -> str:
CampaignRepository(db_session).create(_campaign())
repo = ExplorationSessionRepository(db_session)
session_obj = repo.create(
ExplorationSession(
campaign_id="camp-1",
target_id="t-1",
persona={"name": "急性子用户"},
goal="查询账单并缴费",
)
)
session_obj.status = ExplorationSessionStatus.COMPLETED
repo.update(session_obj)
message_repo = ExplorationMessageRepository(db_session)
for i in range(1, rounds + 1):
message_repo.save_message(
ExplorationMessage(session_id=session_obj.id, round_index=i, role="user", content=f"用户消息-{i}")
)
message_repo.save_message(
ExplorationMessage(session_id=session_obj.id, round_index=i, role="assistant", content=f"回复内容-{i}")
)
return session_obj.id
@pytest.fixture()
def judge_env(db_session, monkeypatch):
from agenteval.exploration import judge as judge_module
monkeypatch.setattr(judge_module, "get_session", lambda: db_session)
return db_session
# ── 抽样 ──────────────────────────────────────────────────────────────
def test_sample_round_indexes_returns_all_within_cap():
assert sample_round_indexes([1, 2]) == [1, 2]
assert sample_round_indexes([1, 2, 3]) == [1, 2, 3]
def test_sample_round_indexes_spreads_evenly_over_cap():
assert sample_round_indexes([1, 2, 3, 4, 5]) == [1, 3, 5]
assert len(sample_round_indexes(list(range(1, 11)))) == MAX_JUDGE_SAMPLES
# ── 归一化 ────────────────────────────────────────────────────────────
def test_normalize_judge_review_applies_whitelists():
raw = {
"dimensions": [
{"dimension": "attitude", "rating": "good", "comment": "ok"},
{"dimension": "神秘维度", "rating": "good", "comment": "drop me"},
{"dimension": "hallucination", "rating": "离谱", "comment": "bad rating"},
],
"summary": "结论",
}
review = normalize_judge_review(raw)
dims = {d["dimension"]: d for d in review["dimensions"]}
assert set(dims) == {"attitude", "hallucination"}
assert dims["attitude"]["rating"] == "good"
assert dims["hallucination"]["rating"] == "acceptable" # 非法档位归一
assert review["summary"] == "结论"
def test_normalize_judge_review_rejects_non_list_dimensions():
review = normalize_judge_review({"dimensions": "不是列表", "summary": 123})
assert review["dimensions"] == []
assert review["summary"] == "123"
# ── 后台执行编排 ──────────────────────────────────────────────────────
async def test_execute_persists_structured_review(judge_env):
session_id = _seed_session_with_messages(judge_env, rounds=5)
_seed_config(judge_env)
client = FakeChatClient(REVIEW_JSON)
await execute_judge_review(session_id, chat_client=client)
session_obj = ExplorationSessionRepository(judge_env).get(session_id)
assert session_obj.status == ExplorationSessionStatus.COMPLETED
review = session_obj.judge_review
assert review["status"] == "completed"
assert review["model_config_id"] == "mc-default"
assert review["sampled_rounds"] == [1, 3, 5]
assert len(review["dimensions"]) == 3
assert review["summary"] == "服务态度好但存在幻觉"
async def test_execute_prompt_carries_only_sampled_rounds(judge_env):
session_id = _seed_session_with_messages(judge_env, rounds=5)
_seed_config(judge_env)
client = FakeChatClient(REVIEW_JSON)
await execute_judge_review(session_id, chat_client=client)
payload = json.dumps(client.calls[0], ensure_ascii=False)
assert "用户消息-1" in payload and "用户消息-3" in payload and "用户消息-5" in payload
assert "用户消息-2" not in payload and "用户消息-4" not in payload
async def test_execute_records_error_on_unparseable_output(judge_env):
session_id = _seed_session_with_messages(judge_env)
_seed_config(judge_env)
await execute_judge_review(session_id, chat_client=FakeChatClient("这不是 JSON"))
session_obj = ExplorationSessionRepository(judge_env).get(session_id)
assert session_obj.status == ExplorationSessionStatus.COMPLETED # 复核失败不阻塞会话
review = session_obj.judge_review
assert review["status"] == "failed"
assert review["error"]
assert review["model_config_id"] == "mc-default"
async def test_execute_records_error_on_client_exception(judge_env):
session_id = _seed_session_with_messages(judge_env)
_seed_config(judge_env)
await execute_judge_review(session_id, chat_client=FakeChatClient(RuntimeError("模型网关超时")))
review = ExplorationSessionRepository(judge_env).get(session_id).judge_review
assert review["status"] == "failed"
assert "模型网关超时" in review["error"]
async def test_execute_skips_silently_without_model_config(judge_env):
session_id = _seed_session_with_messages(judge_env)
client = FakeChatClient(REVIEW_JSON)
await execute_judge_review(session_id, chat_client=client)
assert client.calls == []
assert ExplorationSessionRepository(judge_env).get(session_id).judge_review is None
async def test_execute_skips_session_without_messages(judge_env):
CampaignRepository(judge_env).create(_campaign())
session_obj = ExplorationSessionRepository(judge_env).create(
ExplorationSession(campaign_id="camp-1", target_id="t-1", persona={}, goal="g")
)
_seed_config(judge_env)
client = FakeChatClient(REVIEW_JSON)
await execute_judge_review(session_obj.id, chat_client=client)
assert client.calls == []
assert ExplorationSessionRepository(judge_env).get(session_obj.id).judge_review is None

View File

@ -0,0 +1,106 @@
"""Exploration summary aggregation (v0.9 票据 05).
体验记录聚合成探索摘要会话数目标达成率问题清单blockers/misled
按出现次数降序judge 复核结论已就位才纳入findings 全量收各档发现
poor 档排前缺则无痕无会话返回 None纯函数不落库
"""
from agenteval.exploration.models import ExplorationSession, ExplorationSessionStatus
from agenteval.exploration.summary import summarize_exploration
def _session(experience=None, judge_review=None, status=ExplorationSessionStatus.COMPLETED) -> ExplorationSession:
return ExplorationSession(
campaign_id="c-1",
target_id="t-1",
persona={"name": "x"},
goal="查询账单",
status=status,
experience=experience,
judge_review=judge_review,
)
def _experience(goal_achieved: bool, blockers=None, misled=None) -> dict:
return {
"goal_achieved": goal_achieved,
"blockers": blockers or [],
"misled": misled or [],
"emotion": "neutral",
"notes": "",
}
def test_no_sessions_returns_none():
assert summarize_exploration([]) is None
def test_aggregates_counts_rate_and_issue_lists():
sessions = [
_session(_experience(True, blockers=["缴费入口难找"])),
_session(_experience(False, blockers=["缴费入口难找", "验证码收不到"], misled=["被误导选了错误套餐"])),
_session(_experience(False, blockers=["验证码收不到"])),
]
summary = summarize_exploration(sessions)
assert summary["session_count"] == 3
assert summary["sessions_with_experience"] == 3
assert summary["goal_achieved_count"] == 1
assert summary["goal_achievement_rate"] == round(1 / 3, 4)
# 问题清单按出现次数降序(同频按首次出现顺序)
assert summary["issues"] == [
{"issue": "缴费入口难找", "count": 2},
{"issue": "验证码收不到", "count": 2},
]
assert summary["misled"] == [{"issue": "被误导选了错误套餐", "count": 1}]
def test_sessions_without_experience_excluded_from_rate_denominator():
sessions = [
_session(_experience(True)),
_session(), # 未关闭/无体验记录
]
summary = summarize_exploration(sessions)
assert summary["session_count"] == 2
assert summary["sessions_with_experience"] == 1
assert summary["goal_achievement_rate"] == 1.0
assert summary["issues"] == []
def test_no_experience_at_all_rate_is_none():
summary = summarize_exploration([_session(), _session()])
assert summary["goal_achievement_rate"] is None
assert summary["issues"] == [] and summary["misled"] == []
def test_judge_review_included_only_when_completed():
good_review = {
"status": "completed",
"dimensions": [
{"dimension": "attitude", "rating": "good", "comment": "友好"},
{"dimension": "professionalism", "rating": "acceptable", "comment": "基本准确"},
{"dimension": "hallucination", "rating": "poor", "comment": "编造了政策"},
],
"summary": "存在幻觉",
}
failed_review = {"status": "failed", "error": "解析失败"}
sessions = [
_session(_experience(True), judge_review=good_review),
_session(_experience(False), judge_review=failed_review),
_session(_experience(True)), # 未复核
]
summary = summarize_exploration(sessions)
judge = summary["judge_review"]
assert judge["reviewed_sessions"] == 1
# 复核发现全量纳入poor 档排前(票据 05复核结论一并纳入
assert judge["findings"] == [
{"dimension": "hallucination", "rating": "poor", "comment": "编造了政策"},
{"dimension": "professionalism", "rating": "acceptable", "comment": "基本准确"},
{"dimension": "attitude", "rating": "good", "comment": "友好"},
]
# 复核总体结论一并纳入
assert judge["summaries"] == ["存在幻觉"]
def test_judge_review_absent_when_none_reviewed():
summary = summarize_exploration([_session(_experience(True))])
assert summary["judge_review"] is None

View File

@ -346,3 +346,70 @@ def test_render_campaign_markdown_comparison_comes_after_analysis():
def test_render_campaign_markdown_without_comparison_unchanged():
md = render_campaign_markdown(_campaign_report())
assert "周期对比" not in md
# ── render_campaign_markdown: 探索发现附录v0.9 票据 05────────────────────
def _exploration() -> dict:
return {
"session_count": 3,
"sessions_with_experience": 3,
"goal_achieved_count": 1,
"goal_achievement_rate": 0.3333,
"issues": [
{"issue": "缴费入口难找", "count": 2},
{"issue": "验证码收不到", "count": 1},
],
"misled": [{"issue": "被误导选了错误套餐", "count": 1}],
"judge_review": {
"reviewed_sessions": 1,
"findings": [{"dimension": "hallucination", "rating": "poor", "comment": "编造了不存在的政策"}],
"summaries": ["服务态度好但存在幻觉"],
},
}
def test_render_campaign_markdown_appends_exploration_appendix():
md = render_campaign_markdown(_campaign_report(), exploration=_exploration())
assert "## 探索发现" in md
assert "| 探索会话数 | 3 |" in md
assert "| 目标达成率 | 33.3% |" in md
assert "缴费入口难找 ×2" in md
assert "验证码收不到 ×1" in md
assert "被误导选了错误套餐 ×1" in md
assert "编造了不存在的政策" in md
assert "复核结论:服务态度好但存在幻觉" in md
def test_render_campaign_markdown_exploration_without_judge_or_issues():
md = render_campaign_markdown(
_campaign_report(),
exploration={
"session_count": 1,
"sessions_with_experience": 1,
"goal_achieved_count": 1,
"goal_achievement_rate": 1.0,
"issues": [],
"misled": [],
"judge_review": None,
},
)
assert "## 探索发现" in md
assert "" in md # 空问题清单回落文案
def test_render_campaign_markdown_exploration_comes_after_comparison():
md = render_campaign_markdown(
_campaign_report(), analysis=_analysis(), comparison=_comparison(), exploration=_exploration()
)
assert md.index("## 智能分析") < md.index("## 周期对比") < md.index("## 探索发现")
def test_render_campaign_markdown_without_exploration_byte_identical():
base = render_campaign_markdown(_campaign_report(), analysis=_analysis(), comparison=_comparison())
with_none = render_campaign_markdown(
_campaign_report(), analysis=_analysis(), comparison=_comparison(), exploration=None
)
assert base == with_none
assert "探索发现" not in base