Compare commits

...

5 Commits

Author SHA1 Message Date
sinohqb
3e485bfbe6 docs(v0.9): mark ticket 03 patrol API done
Some checks failed
CI / test (push) Failing after 43s
2026-08-03 18:12:32 +08:00
sinohqb
6340ec503c feat(exploration): stateless patrol API with watermark increments
Resident agents call GET /api/exploration/patrol once per cycle to see
every running production-line campaign that opted into exploration
(seed set present), the new results since the last watermark (reusing
campaign report aggregation), and the remaining exploration budget.
The watermark advances after each call so subsequent calls only report
increments; accelerated and terminal campaigns are excluded.
2026-08-03 18:12:11 +08:00
sinohqb
53afb9b5d1 feat(campaigns): exploration seed set and budget config per campaign
v0.9 ticket 02. Campaigns now carry an exploration seed set (seed
personas × seed goals — the comparability unit for exploratory
evaluation) and an optional budget override, stored as JSON columns
isomorphic to plan. Empty seeds normalize to null, marking the campaign
as opted out of exploration. resolve_budget merges per-field overrides
into platform defaults; enforcement stays server-side. The create form
gains seed lists and budget inputs (minutes → seconds), submitting null
when left empty.
2026-08-03 17:52:25 +08:00
sinohqb
a351f65550 feat(exploration): session lifecycle endpoints with platform hard guardrails
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
2026-08-03 17:35:17 +08:00
sinohqb
c35159ffcb docs(v2): record exploratory-evaluation consensus in CONTEXT.md and ADR-0003
Grilling 沉淀:OpenClaw 定位从"重规划"转为"虚拟用户"双轨结构
(周期活动内嵌探索先行 + 探索活动后续里程碑),新增六个领域词条。
2026-08-03 16:40:28 +08:00
29 changed files with 1841 additions and 10 deletions

View File

@ -0,0 +1,13 @@
# 01 — 探索会话生命周期 + 平台硬护栏
**What to build:** 从 API 视角完整走通一段探索会话:指定活动、人设与目标创建 running 会话 → 以虚拟用户身份发消息并收到被评对象回复(经目标通道转发,双方轮次持久化、记录延迟)→ 提交结构化体验记录(达成、卡点、被误导、情绪)关闭会话。护栏是平台账本:超窗口会话数、超单会话轮数、相邻会话间隔不足,一律 409 并附原因;会话非 running 状态拒收消息。探索会话是独立实体,不并入评测运行(不污染通过率口径)。
**Blocked by:** None — can start immediately
**Status:** completed
- [x] 探索会话与会话轮次独立建表,迁移可在全新库与既有库上应用
- [x] 创建/发消息/关闭三个端点全链路可用X-API-Key 鉴权
- [x] 三类预算超限各返回 409 + 可读原因平台默认值生效≤8 会话/窗口、≤12 轮/会话、≥30min 间隔)
- [x] 体验记录结构校验,非法值归一(沿 v0.7 白名单经验)
- [x] 集成测试覆盖完整生命周期与各拒绝分支(先例:活动 API 集成测试)

View File

@ -0,0 +1,13 @@
# 02 — 活动级种子集与探索预算配置
**What to build:** 创建活动时可为该活动配置种子集(种子人设数组 + 种子目标数组)与探索预算(最大会话数 / 单会话最大轮数 / 会话最小间隔,空则用平台默认)。配置与活动计划同构入库,可查询;前端活动创建表单提供种子与预算输入,种子留空即该活动不参与探索。种子集是探索式评测的可比性单位,随活动持久化。
**Blocked by:** None — can start immediately
**Status:** completed
- [x] 活动实体新增种子集与预算两个 JSON 配置字段,迁移可在全新库与既有库上应用
- [x] 活动创建/查询 API 接受并返回配置;预算空值回落平台默认
- [x] 创建表单可录入种子人设、种子目标与预算覆盖
- [x] 种子留空的活动创建成功且标记为不参与探索
- [x] 集成测试覆盖配置存取与默认回落

View File

@ -0,0 +1,12 @@
# 03 — 巡检 API
**What to build:** 常驻代理一次调用即可无状态巡检返回所有进行中的正式线活动time_scale == 1清单每项含活动标识与对象、自上次巡检以来的新结果摘要复用现有活动报告聚合口径不重算新指标、探索预算余量剩余会话数等。响应生成后推进各活动的巡检水位使下次调用只报增量。加速线活动不出现在巡检结果中。
**Blocked by:** 01预算余量需会话计数、02活动预算字段
**Status:** completed
- [x] 巡检端点返回进行中正式线活动 + 新结果摘要 + 预算余量
- [x] 巡检水位推进:连续两次调用,第二次只含增量
- [x] 加速线与终态活动不出现
- [x] 集成测试覆盖内容正确性、水位推进与过滤规则

View File

@ -0,0 +1,12 @@
# 04 — judge 抽样复核
**What to build:** 会话关闭后,平台后台对该会话对话抽样(默认 ≤3 段,控 token经 judge 岗位模型独立复核产出质量维度结论态度、专业性、幻觉等落入会话记录。复核是异步后台执行失败落错误不阻塞会话状态也不影响体验记录这条第一手证据线。LLM 调用走可注入客户端接缝,测试用假客户端覆盖。
**Blocked by:** 01会话关闭事件与对话数据
**Status:** ready-for-agent
- [ ] 会话关闭后自动触发抽样复核,结果结构化落库
- [ ] 抽样上限生效;无模型配置时静默跳过
- [ ] 复核失败落错误,会话仍为 completed
- [ ] 假客户端测试覆盖编排、解析失败、无模型分支先例v0.7 分析测试)

View File

@ -0,0 +1,12 @@
# 05 — 探索发现 → 报告 / 分析 / 导出
**What to build:** 探索数据汇入既有报告链的三个出口。活动报告新增"探索发现"维度会话数、目标达成率、问题清单来自体验记录聚合judge 复核结论若已就位一并纳入。v0.7 活动分析的输入追加探索摘要(问题清单 + 达成统计非全量对话让分析模型合成两条证据线。Markdown 导出在存在探索数据时追加「探索发现」附录(顺序在智能分析与周期对比之后),无探索数据时导出与旧版完全一致。周期对比口径不变。
**Blocked by:** 01聚合主料是体验记录04 若已就位则复核结论一并纳入,未就位不阻塞
**Status:** ready-for-agent
- [ ] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
- [ ] 分析输入含探索摘要且不含全量对话
- [ ] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
- [ ] 纯函数渲染与聚合单测 + 导出集成测试先例report_render 单测、活动导出测试)

View File

@ -0,0 +1,12 @@
# 06 — 报告抽屉「探索发现」区块 + 会话下钻
**What to build:** 活动报告抽屉新增「探索发现」区块(模式沿 v0.7 智能分析区块):会话数、目标达成率、问题清单、会话列表状态。每个会话可点开查看完整对话与体验记录——证据下钻是探索数据的命脉,"用户在这里被绕晕"必须能看到原文。无探索数据时区块不出现。不建新顶级页面。
**Blocked by:** 05报告聚合与查询端点就绪
**Status:** ready-for-agent
- [ ] 报告抽屉呈现探索发现区块,数据与后端口径一致
- [ ] 会话详情展示完整对话(角色、内容、延迟)与体验记录
- [ ] 无探索数据时区块隐藏
- [ ] tsc 通过;人工在浏览器验证展开/下钻交互

View File

@ -0,0 +1,12 @@
# 07 — 活动 finalize 会话结算 + 档位校验
**What to build:** 活动生命周期与探索会话的收口规则。窗口结束进入终态结算时,仍 running 的探索会话转 expired此后拒收消息——不留悬挂会话。档位校验完整化正式线接受自动与手动触发的会话加速调试线仅接受手动触发自动来源一律拒绝时间压缩与拟真冲突加速线不参与自动探索
**Blocked by:** 01会话实体与状态机
**Status:** ready-for-agent
- [ ] 活动终态结算时 running 会话转 expired后续消息 409
- [ ] 加速线自动触发被拒并附原因;手动触发可用
- [ ] 正式线两种触发来源均可用
- [ ] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)

View File

@ -0,0 +1,12 @@
# 08 — AI 助手巡检/探索 skill + 常驻作业
**What to build:** 让已集成的 AI 助手成为常驻巡检代理:编写巡检/探索 skill说明书指导代理按"调巡检 API → 研判新结果与预算余量 → 决定是否派发探索会话 → 驱动会话对话 → 提交体验记录"的闭环行动,含收到 409 时的收敛行为与衍生变体的种子回溯要求。skill 走部署脚本既有的 skill 同步管线分发;在 AI 助手上注册一个全局常驻 heartbeat/cron 巡检作业(默认节拍 1h。在 t480 端到端演示:一个配置了种子集的正式线活动,在窗口内被自动巡检并产出探索会话。
**Blocked by:** 03巡检 API 契约就绪才能写说明书)
**Status:** ready-for-agent
- [ ] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
- [ ] 部署同步管线分发 skill复用现有同步步骤
- [ ] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
- [ ] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库

113
.scratch/v0.9/spec.md Normal file
View File

@ -0,0 +1,113 @@
# v0.9 规格说明:探索式评测第一期 —— 周期活动内嵌虚拟用户探索
**状态**: ready-for-agent
**日期**: 2026-08-03
**决策依据**: ADR-0003 v2 修订2026-08-03、CONTEXT.md「探索式评测」章节、grilling 共识清单15 项)
## Problem Statement
平台的评测能力目前全部建立在"固定场景(考纲)"之上:预设题目、预设规则、统计通过率。这能回答"考纲过了多少",回答不了"真实用户会踩到什么"——真实用户不按考纲出牌,他们会中途改主意、追问、被绕晕、放弃。被评智能体在面对这类复杂拟真行为时的表现(问题诊断与优化建议)目前没有任何评测手段。
## Solution
引入与固定场景并行的**探索式评测**模式OpenClaw 作为**虚拟用户**,按活动配置的种子人设 × 种子目标自主与被评对象对话,产出**体验记录**目标达成、卡点、被误导处judge 岗位抽样复核两条证据线汇入活动报告、v0.7 分析与 Markdown 导出。
第一期形态为**周期活动内嵌探索**:现有活动骨架不变(静态计划照常执行),全局一个常驻巡检代理(跑在已集成的 AI 助手上heartbeat/cron 唤醒按节拍巡检进行中的正式线活动在预算内派发探索会话。OpenClaw 停摆只暂停探索,固定计划照常完成——耐久性归平台,自主性归代理。
## User Stories
1. As an 评测平台用户, I want 创建活动时配置该活动的种子集(种子人设 + 种子目标), so that 虚拟用户的行动有据可依,且同种子集的活动跨期可比
2. As an 评测平台用户, I want 创建活动时调整探索预算(或使用平台默认值), so that 探索行为的规模与负载在我的掌控之内
3. As an 平台运维者, I want 部署时 AI 助手注册一个全局常驻巡检作业, so that 所有进行中的正式线活动被自动巡检,无需逐活动配置
4. As an 常驻代理, I want 每次巡检唤醒时调平台巡检 API 拿到进行中活动、上次巡检以来的新结果与预算余量, so that 我能无状态地自主决定"继续观察 / 派探索会话"
5. As an 常驻代理, I want 通过 API 创建探索会话(指定活动、人设、目标、种子出处), so that 我能以虚拟用户身份开始行动
6. As an 常驻代理, I want 通过会话 API 发消息并收到被评对象的回复, so that 我能按人设自主决定追问、等待、放弃
7. As an 常驻代理, I want 在会话结束时提交结构化体验记录, so that 我的第一手判定(达成/卡点/被误导)被平台留痕
8. As an 常驻代理, I want 超出预算时收到 API 明确的 409 拒绝与原因, so that 拒绝本身成为反馈信号,我及时收敛
9. As an 常驻代理, I want 在预算内从种子衍生变体(同目标换表达、同人设换追问策略)并被标记衍生出处, so that 行为多样性与留痕兼得
10. As an 评测平台用户, I want 在活动报告抽屉看到"探索发现"区块(会话数、目标达成率、问题清单), so that 我知道真实用户遭遇了什么
11. As an 评测平台用户, I want 点开单个探索会话查看完整对话与体验记录, so that 每个"问题"都能下钻到原文证据
12. As an 分析岗位模型, I want 生成活动分析时收到探索摘要(问题清单 + 达成统计,非全量对话), so that "固定场景成绩"与"真实用户遭遇"两条证据线合成一份结论
13. As an judge 岗位模型, I want 对抽样的探索对话做独立复核, so that 体验判定之外有客观质量维度(态度、专业性、幻觉)
14. As an 报告导出用户, I want Markdown 导出在存在探索数据时追加"探索发现"区块, so that 离线分享材料完整;无探索数据时导出与旧版一致
15. As an 加速调试线用户, I want 手动触发探索会话, so that 我能调试探索流程;同时加速线不被自动巡检打扰(时间压缩与拟真冲突)
16. As an 平台, I want OpenClaw 停摆时活动的固定计划照常走完, so that 探索是增强层而非单点故障
17. As an 平台, I want 活动窗口结束时仍未关闭的探索会话被妥善结算(标记状态、不再接受消息), so that 不留悬挂会话
18. As an 评测平台用户, I want 探索会话与固定场景的子 Run 在报告里清楚区分, so that 通过率口径ADR-0002不被探索数据污染
## Implementation Decisions
### 数据模型
- 新增**探索会话**独立实体不并入评测运行归属活动、指向评测对象、人设JSON、目标、种子出处可空 = 衍生变体须回溯到种子、状态机running / completed / failed / expired、触发来源auto / manual、体验记录 JSON、judge 复核 JSON、轮次计数、时间戳
- 新增会话轮次表role、content、latency_ms、created_at与评测运行的 turns 表平行,不复用
- 活动实体新增两个 JSON 配置字段(与 `plan` 同构地位):**种子集**(种子人设数组 + 种子目标数组)、**探索预算**(最大会话数 / 单会话最大轮数 / 会话最小间隔,空则取平台默认)
- 活动实体新增 `last_patrolled_at`(巡检 API 每次读取后更新,支撑"上次巡检以来的新结果"语义)
- 三个变更各走一个 Alembic 迁移batch mode
### API 契约(全部 X-API-Key 鉴权,沿用 `require_api_key`
- `GET /api/exploration/patrol`返回进行中的正式线活动time_scale == 1清单每项含活动 id/名称/对象、自 `last_patrolled_at` 以来的新结果摘要(复用 `build_campaign_report` 聚合口径)、探索预算余量;响应生成后更新各活动 `last_patrolled_at`
- `POST /api/exploration/sessions`body = campaign_id、persona、goal、seed_ref可空平台硬校验活动存在且 running、正式线仅接受 auto 或 manual 触发;加速线仅接受 manual预算余量→ 创建 running 会话;超限 409 + 原因
- `POST /api/exploration/sessions/{id}/messages`body = content平台转发到目标的通道复用 ChannelFactory、持久化双方轮次、返回回复与延迟单会话轮数超限 409会话非 running 状态 409
- `POST /api/exploration/sessions/{id}/close`body = 体验记录goal_achieved: bool、blockers[]、misled[]、emotion、notes结构校验 + 非法值归一(沿 v0.7 白名单经验);会话转 completed
- `GET /api/campaigns/{id}/exploration`:会话列表 + 探索发现聚合(会话数、达成率、问题清单)
- 活动窗口 finalize 时:仍 running 的会话转 expired不再接受消息`resolve_finalize` 同处挂接)
### 护栏(平台硬执行,不信任客户端自律)
- 平台默认≤8 会话/窗口、≤12 轮/会话、相邻会话 ≥30min正式线真实时间活动级预算覆盖
- 预算计数是平台账本:创建/发消息/关闭均实时校验,超限一律 409
### 判定双证据线
- 体验判定close 接口收结构化自报,为第一手证据
- judge 抽样复核:会话结束后平台对对话抽样(默认 ≤3 段,控 token经 judge 岗位模型产出质量维度复核ChatClient 可注入(沿 v0.7 分析 seam异步后台执行失败落错误不阻塞
### 报告 / 分析 / 导出
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);周期对比口径不变(探索数据不参与)
- v0.7 分析输入追加探索**摘要**(问题清单 + 达成统计,非全量对话)
- Markdown 导出:存在探索数据时追加「探索发现」附录(缺则无痕),附录顺序:智能分析 → 周期对比 → 探索发现
### 前端
- 活动创建表单:种子集与预算配置输入(种子可留空 = 该活动不参与探索)
- 报告抽屉新增「探索发现」区块(模式沿 v0.7 智能分析区块):达成率、问题清单、会话列表;会话点开查看完整对话 + 体验记录
- 不建新顶级页面
### AI 助手侧(不在平台代码仓内的行为,契约即平台 API
- 现有已集成 OpenClawAI 助手菜单、openclaw-eval 容器)上扩展巡检/探索 skill走部署脚本既有 skill 同步管线
- 注册一个全局常驻 heartbeat/cron 巡检作业(节拍默认 1h巡检 API 本身与节拍无关
### 档位
- 正式线time_scale == 1自动巡检 + 自动派发
- 加速调试线:仅手动触发会话,不参与自动巡检
## Testing Decisions
- 好测试只测外部行为API 契约、状态机迁移、护栏拒绝、聚合口径、渲染产物;不测内部编排细节
- **集成测试TestClient主力接缝**:会话生命周期(创建→对话→关闭)、护栏 409 各分支(超会话数/超轮数/间隔不足/加速线 auto 拒绝)、巡检 API 内容与 last_patrolled_at 推进、finalize 结算 expired、报告/导出纳入。先例:`test_campaigns_api.py`、`test_campaign_comparison_api.py`
- **纯函数单测**:探索发现聚合 + Markdown 渲染dict 进 string 出)。先例:`test_report_render.py`
- **假客户端测试**judge 抽样复核与分析输入扩展,注入假 ChatClient。先例`test_campaign_analysis.py`
- 前端仅 `tsc --noEmit`(无 vitest沿用现状
- AI 助手 skill 行为不进自动化测试,靠部署后端到端验证(与 v0.4 以来 agenteval-run skill 验收方式一致)
## Out of Scope
- 双轨之二「探索活动」独立活动类型OpenClaw 全权接管生命周期)——后续里程碑
- 周期对比扩展(探索数据跨期对照)——待种子集指纹与达成率口径稳定
- 计划条目重排(原 ADR-0003 v2 字面意义的"自适应重规划")——本期代理只派发探索会话,不改写静态计划
- 全局种子库/人设库(活动级配置已够,复用需求出现再提取)
- 事件驱动唤醒(纯 heartbeat/cron 巡检)
- 前端测试基线vitest
- volcengine-102 同步
## Further Notes
- 词汇表与 ADR 已先行落地commit c35159fCONTEXT.md「探索式评测」章节、ADR-0003 v2 修订
- 巡检 API 的"新结果摘要"复用 `build_campaign_report` 聚合口径,不重算新指标
- OpenClaw heartbeat/cron 事实依据:作业持久化 SQLite、重启不丢、瞬态错误重试、连续 10 次失败自动禁用——天然熔断,与"决策点不可用则跳过"的可靠性边界吻合
- 里程碑目录 `.scratch/v0.9/`,票据由 `/to-tickets` 生成于 `issues/`

View File

@ -82,6 +82,32 @@ _Avoid_: 环比(暗示固定自然周期,活动窗口可任意长)
活动的两种运行档位,由时间倍速区分:`time_scale == 1` 为正式线(窗口按真实时长走完,代表真实服务周期,享受自动分析、自动周期对比等全链自动化);`time_scale > 1` 为加速调试线(压缩窗口用于快速验证流程,自动化能力默认关闭,仅手动按需触发)。
_Avoid_: 真实线、快速模式
## 探索式评测
**虚拟用户Virtual User**:
OpenClaw 在探索式评测中扮演的角色:按人设与探索目标模拟拟真用户行为、与被评对象自主对话的智能体。是"行动者"而非"台词生成器"——对话节奏、追问、放弃均由其自主决定。
_Avoid_: 测试员、脚本机器人、模拟器
**探索式评测Exploratory Evaluation**:
与固定场景并行的第二种评测模式:不预设考纲,由虚拟用户按种子自由行动,产出"发现的问题"而非"通过率"。固定场景保留为回归基线(可比性、周期对比建在其上),两模式共用活动/窗口/报告外壳,可在同一活动内混编。(决策见 ADR-0003 v2 修订)
_Avoid_: 自由测试、压力测试、探索测试
**探索会话Exploration Session**:
虚拟用户以"人设 × 目标"组合执行的一段完整对话过程,独立实体存储(`exploration_sessions`不并入评测运行——Run 绑定场景与规则判定,探索会话两者皆无)。平台提供会话对象、护栏与留痕,对话由 OpenClaw 驱动。
_Avoid_: 运行、用例、测试会话
**种子集Seed Set**:
活动级配置的种子人设(背景、性格、耐心度)与种子目标(如"完成退货")集合,是探索式评测的可比性单位(对应固定场景的"考纲",与 `plan` 同构)。虚拟用户可在预算约束内从种子衍生有限变体(换表达、换追问策略),全部留痕。
_Avoid_: 题库、人设库、剧本
**体验记录Experience Record**:
探索会话结束时虚拟用户产出的结构化自报:目标是否达成、卡点、被误导处、情绪变化。探索式评测的第一手证据——意图-结果闭环只有行动者自己能给judge 岗位对抽样对话独立复核补充质量维度,两条证据链在分析层汇合。
_Avoid_: 评测结果、断言、日志
**巡检Patrol**:
常驻代理的周期性自主行动:经 OpenClaw heartbeat/cron 唤醒,调平台巡检 API 查看进行中活动的新结果与预算余量,自主决定"继续观察 / 派探索会话"。全局一个常驻巡检作业无状态地巡检所有活动OpenClaw 停摆只暂停探索,固定计划照常。
_Avoid_: 轮询、心跳heartbeat 是 OpenClaw 机制名,巡检是平台侧行为)
## 模型配置
**模型能力Capability**:

View File

@ -0,0 +1 @@
"""Exploratory evaluation (探索式评测): virtual-user sessions inside campaigns."""

View File

@ -0,0 +1,107 @@
"""Domain models and budget defaults for exploratory evaluation (v0.9).
Exploration sessions are an independent entity never merged into EvalRun
so pass-rate semantics (ADR-0002) and scenario comparability (ADR-0001) stay
untouched. Budget enforcement is a platform ledger: the defaults here apply
unless overridden per-campaign via ``Campaign.exploration_budget``; the
enforcement itself always happens server-side.
"""
from datetime import datetime
from enum import Enum
from typing import Any, Optional
from pydantic import BaseModel, Field
from agenteval.models import Campaign
DEFAULT_MAX_SESSIONS_PER_WINDOW = 8
DEFAULT_MAX_TURNS_PER_SESSION = 12
DEFAULT_MIN_SESSION_INTERVAL_SECONDS = 30 * 60
VALID_EMOTIONS = ("positive", "neutral", "confused", "frustrated")
class ExplorationSessionStatus(str, Enum):
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
EXPIRED = "expired"
class ExplorationTrigger(str, Enum):
AUTO = "auto"
MANUAL = "manual"
class ExplorationMessage(BaseModel):
"""One chat message (role/content) inside an exploration session."""
id: Optional[str] = None
session_id: str
round_index: int = 0
role: str = "user"
content: str = ""
latency_ms: Optional[int] = None
created_at: Optional[datetime] = None
class ExplorationSession(BaseModel):
"""A virtual-user exploration session belonging to one campaign."""
id: Optional[str] = None
campaign_id: str
target_id: str
persona: dict[str, Any] = Field(default_factory=dict)
goal: str = ""
seed_ref: Optional[dict[str, Any]] = None
status: ExplorationSessionStatus = ExplorationSessionStatus.RUNNING
triggered_by: ExplorationTrigger = ExplorationTrigger.AUTO
experience: Optional[dict[str, Any]] = None
judge_review: Optional[dict[str, Any]] = None
turn_count: int = 0
error: Optional[str] = None
created_at: Optional[datetime] = None
closed_at: Optional[datetime] = None
class ExplorationBudget(BaseModel):
max_sessions: int = DEFAULT_MAX_SESSIONS_PER_WINDOW
max_turns: int = DEFAULT_MAX_TURNS_PER_SESSION
min_interval_seconds: int = DEFAULT_MIN_SESSION_INTERVAL_SECONDS
def resolve_budget(campaign: Campaign) -> ExplorationBudget:
"""Effective exploration budget for a campaign.
Platform defaults with per-field campaign overrides; unset override
fields fall back to the defaults.
"""
override = campaign.exploration_budget
if override is None:
return ExplorationBudget()
return ExplorationBudget(
max_sessions=override.max_sessions or DEFAULT_MAX_SESSIONS_PER_WINDOW,
max_turns=override.max_turns or DEFAULT_MAX_TURNS_PER_SESSION,
min_interval_seconds=override.min_interval_seconds or DEFAULT_MIN_SESSION_INTERVAL_SECONDS,
)
def normalize_experience(raw: dict[str, Any]) -> dict[str, Any]:
"""Whitelist-normalize a self-reported experience record (v0.7 precedent).
Invalid values are coerced to safe defaults rather than rejected, so a
sloppy virtual user still leaves a usable evidence row.
"""
raw_blockers = raw.get("blockers") if isinstance(raw.get("blockers"), list) else []
raw_misled = raw.get("misled") if isinstance(raw.get("misled"), list) else []
blockers = [str(item) for item in raw_blockers if isinstance(item, (str, int, float))]
misled = [str(item) for item in raw_misled if isinstance(item, (str, int, float))]
emotion = raw.get("emotion")
return {
"goal_achieved": bool(raw.get("goal_achieved")),
"blockers": blockers,
"misled": misled,
"emotion": emotion if emotion in VALID_EMOTIONS else "neutral",
"notes": str(raw.get("notes") or ""),
}

View File

@ -271,6 +271,27 @@ class CampaignSummary(BaseModel):
scheduler: SchedulerState = Field(default_factory=SchedulerState)
class ExplorationSeeds(BaseModel):
"""Seed set (种子集) for exploratory evaluation: seed personas × seed goals.
The comparability unit for exploratory evaluation campaigns sharing a
seed set are comparable across periods. Empty/absent means the campaign
opts out of exploration.
"""
personas: list[str] = Field(default_factory=list)
goals: list[str] = Field(default_factory=list)
class ExplorationBudgetConfig(BaseModel):
"""Per-campaign exploration budget override; unset fields fall back to
platform defaults at enforcement time."""
max_sessions: Optional[int] = Field(default=None, gt=0)
max_turns: Optional[int] = Field(default=None, gt=0)
min_interval_seconds: Optional[int] = Field(default=None, gt=0)
class Campaign(BaseModel):
"""An evaluation campaign: a service-cycle window over a single target,
driving many child Runs from a static plan (ADR-0003)."""
@ -290,6 +311,9 @@ class Campaign(BaseModel):
created_at: Optional[datetime] = None
summary: Optional[CampaignSummary] = None
analysis_model_config_id: Optional[str] = None
exploration_seeds: Optional[ExplorationSeeds] = None
exploration_budget: Optional[ExplorationBudgetConfig] = None
last_patrolled_at: Optional[datetime] = None
class Turn(BaseModel):

View File

@ -27,6 +27,15 @@ def utc_now() -> datetime:
return datetime.now(timezone.utc)
def as_utc(dt: datetime) -> datetime:
"""Attach UTC tzinfo to a naive datetime.
SQLite round-trips drop tzinfo; stored times are always UTC, so a naive
value read back is restored as UTC before any comparison with utc_now().
"""
return dt if dt.tzinfo is not None else dt.replace(tzinfo=timezone.utc)
def iso_utc(dt: datetime | None) -> str | None:
"""Serialize a datetime to ISO 8601 with UTC timezone suffix.
@ -188,6 +197,9 @@ class CampaignDB(SQLModel, table=True):
created_at: Optional[datetime] = Field(default_factory=utc_now)
summary: Optional[str] = None
analysis_model_config_id: Optional[str] = None
exploration_seeds: Optional[str] = None
exploration_budget: Optional[str] = None
last_patrolled_at: Optional[datetime] = None
def get_plan(self) -> list[dict[str, Any]]:
return _json_loads(self.plan)
@ -201,6 +213,18 @@ class CampaignDB(SQLModel, table=True):
def set_summary(self, summary: dict[str, Any]) -> None:
self.summary = _json_dumps(summary)
def get_exploration_seeds(self) -> Optional[dict[str, Any]]:
return _json_loads(self.exploration_seeds) if self.exploration_seeds else None
def set_exploration_seeds(self, seeds: dict[str, Any]) -> None:
self.exploration_seeds = _json_dumps(seeds)
def get_exploration_budget(self) -> Optional[dict[str, Any]]:
return _json_loads(self.exploration_budget) if self.exploration_budget else None
def set_exploration_budget(self, budget: dict[str, Any]) -> None:
self.exploration_budget = _json_dumps(budget)
class CampaignAnalysisDB(SQLModel, table=True):
"""One row per campaign holding its intelligent analysis (智能分析) state."""
@ -251,6 +275,75 @@ class CampaignPeriodComparisonDB(SQLModel, table=True):
self.result = _json_dumps(result)
class ExplorationSessionDB(SQLModel, table=True):
"""One virtual-user exploration session (探索会话) within a campaign.
Independent entity, deliberately NOT an EvalRun: exploration outcomes feed
the 体验判定 evidence line and must not pollute pass-rate semantics
(ADR-0001 comparability / ADR-0002).
"""
__tablename__ = "exploration_sessions"
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
campaign_id: str = Field(index=True, foreign_key="campaigns.id")
target_id: str = Field(foreign_key="eval_targets.id")
persona: str = "{}"
goal: str = ""
seed_ref: Optional[str] = None
status: str = "running"
triggered_by: str = "auto"
experience: Optional[str] = None
judge_review: Optional[str] = None
turn_count: int = 0
error: Optional[str] = None
created_at: Optional[datetime] = Field(default_factory=utc_now)
closed_at: Optional[datetime] = None
def get_persona(self) -> dict[str, Any]:
return _json_loads(self.persona)
def set_persona(self, persona: dict[str, Any]) -> None:
self.persona = _json_dumps(persona)
def get_seed_ref(self) -> Optional[dict[str, Any]]:
return _json_loads(self.seed_ref) if self.seed_ref else None
def set_seed_ref(self, seed_ref: dict[str, Any]) -> None:
self.seed_ref = _json_dumps(seed_ref)
def get_experience(self) -> Optional[dict[str, Any]]:
return _json_loads(self.experience) if self.experience else None
def set_experience(self, experience: dict[str, Any]) -> None:
self.experience = _json_dumps(experience)
def get_judge_review(self) -> Optional[dict[str, Any]]:
return _json_loads(self.judge_review) if self.judge_review else None
def set_judge_review(self, judge_review: dict[str, Any]) -> None:
self.judge_review = _json_dumps(judge_review)
class ExplorationMessageDB(SQLModel, table=True):
"""One chat message inside an exploration session (role/content form).
Parallel to ``eval_runs`` turns but never shared with them. A completed
question-answer pair contributes two rows (user + assistant); the reply
row carries ``latency_ms``.
"""
__tablename__ = "exploration_messages"
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
session_id: str = Field(index=True, foreign_key="exploration_sessions.id")
round_index: int = 0
role: str = "user"
content: str = ""
latency_ms: Optional[int] = None
created_at: Optional[datetime] = Field(default_factory=utc_now)
class EvalRunDB(SQLModel, table=True):
"""Database table for evaluation runs."""

View File

@ -4,6 +4,7 @@ from typing import Generic, Optional, TypeVar
from sqlmodel import Session, select
from agenteval.exploration.models import ExplorationMessage, ExplorationSession
from agenteval.models import Campaign, Case, EvalResult, EvalRun, EvalTarget, Scenario
from agenteval.services.model_configs import ModelConfigService
from agenteval.storage.db import (
@ -13,6 +14,8 @@ from agenteval.storage.db import (
EvalResultDB,
EvalRunDB,
EvalTargetDB,
ExplorationMessageDB,
ExplorationSessionDB,
ScenarioDB,
TurnDB,
get_session,
@ -282,11 +285,7 @@ class RunRepository(BaseRepository[EvalRun, EvalRunDB]):
)
def list_by_campaign(self, campaign_id: str) -> list[EvalRun]:
statement = (
select(EvalRunDB)
.where(EvalRunDB.campaign_id == campaign_id)
.order_by(EvalRunDB.started_at)
)
statement = select(EvalRunDB).where(EvalRunDB.campaign_id == campaign_id).order_by(EvalRunDB.started_at)
return [self._from_db(r) for r in self.session.exec(statement).all()]
def mark_orphans_failed(self) -> int:
@ -353,10 +352,15 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
completed_at=campaign.completed_at,
created_at=campaign.created_at,
analysis_model_config_id=campaign.analysis_model_config_id,
last_patrolled_at=campaign.last_patrolled_at,
)
db.set_plan([entry.model_dump(mode="json") for entry in campaign.plan])
if campaign.summary:
db.set_summary(campaign.summary.model_dump(mode="json"))
if campaign.exploration_seeds is not None:
db.set_exploration_seeds(campaign.exploration_seeds.model_dump(mode="json"))
if campaign.exploration_budget is not None:
db.set_exploration_budget(campaign.exploration_budget.model_dump(mode="json"))
return db
def _from_db(self, db: CampaignDB) -> Campaign:
@ -373,6 +377,9 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
created_at=db.created_at,
summary=db.get_summary(),
analysis_model_config_id=db.analysis_model_config_id,
exploration_seeds=db.get_exploration_seeds(),
exploration_budget=db.get_exploration_budget(),
last_patrolled_at=db.last_patrolled_at,
)
def update(self, campaign: Campaign) -> Optional[Campaign]:
@ -388,8 +395,13 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
existing.started_at = campaign.started_at
existing.completed_at = campaign.completed_at
existing.analysis_model_config_id = campaign.analysis_model_config_id
existing.last_patrolled_at = campaign.last_patrolled_at
if campaign.summary is not None:
existing.set_summary(campaign.summary.model_dump(mode="json"))
if campaign.exploration_seeds is not None:
existing.set_exploration_seeds(campaign.exploration_seeds.model_dump(mode="json"))
if campaign.exploration_budget is not None:
existing.set_exploration_budget(campaign.exploration_budget.model_dump(mode="json"))
self.session.add(existing)
self.session.commit()
self.session.refresh(existing)
@ -441,9 +453,7 @@ class CampaignPeriodComparisonRepository:
self.session = session or get_session()
def get_by_campaign(self, campaign_id: str) -> Optional[CampaignPeriodComparisonDB]:
statement = select(CampaignPeriodComparisonDB).where(
CampaignPeriodComparisonDB.campaign_id == campaign_id
)
statement = select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == campaign_id)
return self.session.exec(statement).first()
def upsert(
@ -510,3 +520,113 @@ class ResultRepository:
self.session.commit()
self.session.refresh(db)
return _result_from_db(db)
class ExplorationSessionRepository(BaseRepository[ExplorationSession, ExplorationSessionDB]):
"""Repository for virtual-user exploration sessions (探索会话)."""
_table = ExplorationSessionDB
_order_by = "created_at"
def _copy_mutable(self, db: ExplorationSessionDB, session_obj: ExplorationSession) -> None:
db.goal = session_obj.goal
db.status = session_obj.status.value
db.triggered_by = session_obj.triggered_by.value
db.turn_count = session_obj.turn_count
db.error = session_obj.error
db.closed_at = session_obj.closed_at
db.set_persona(session_obj.persona)
if session_obj.seed_ref is not None:
db.set_seed_ref(session_obj.seed_ref)
if session_obj.experience is not None:
db.set_experience(session_obj.experience)
if session_obj.judge_review is not None:
db.set_judge_review(session_obj.judge_review)
def _to_db(self, session_obj: ExplorationSession) -> ExplorationSessionDB:
db = ExplorationSessionDB(
id=session_obj.id,
campaign_id=session_obj.campaign_id,
target_id=session_obj.target_id,
created_at=session_obj.created_at,
)
self._copy_mutable(db, session_obj)
return db
def _from_db(self, db: ExplorationSessionDB) -> ExplorationSession:
return ExplorationSession(
id=db.id,
campaign_id=db.campaign_id,
target_id=db.target_id,
persona=db.get_persona(),
goal=db.goal,
seed_ref=db.get_seed_ref(),
status=db.status,
triggered_by=db.triggered_by,
experience=db.get_experience(),
judge_review=db.get_judge_review(),
turn_count=db.turn_count,
error=db.error,
created_at=db.created_at,
closed_at=db.closed_at,
)
def update(self, session_obj: ExplorationSession) -> Optional[ExplorationSession]:
existing = self.session.get(ExplorationSessionDB, session_obj.id)
if not existing:
return None
self._copy_mutable(existing, session_obj)
self.session.add(existing)
self.session.commit()
self.session.refresh(existing)
return self._from_db(existing)
def list_by_campaign(self, campaign_id: str) -> list[ExplorationSession]:
statement = (
select(ExplorationSessionDB)
.where(ExplorationSessionDB.campaign_id == campaign_id)
.order_by(ExplorationSessionDB.created_at)
)
return [self._from_db(r) for r in self.session.exec(statement).all()]
class ExplorationMessageRepository:
"""Append-only repository for exploration session chat rows."""
def __init__(self, session: Optional[Session] = None):
self.session = session or get_session()
def save_message(self, message: ExplorationMessage) -> ExplorationMessage:
db = ExplorationMessageDB(
id=message.id,
session_id=message.session_id,
round_index=message.round_index,
role=message.role,
content=message.content,
latency_ms=message.latency_ms,
created_at=message.created_at,
)
self.session.add(db)
self.session.commit()
self.session.refresh(db)
message.id = db.id
return message
def list_by_session(self, session_id: str) -> list[ExplorationMessage]:
statement = (
select(ExplorationMessageDB)
.where(ExplorationMessageDB.session_id == session_id)
.order_by(ExplorationMessageDB.created_at)
)
return [
ExplorationMessage(
id=r.id,
session_id=r.session_id,
round_index=r.round_index,
role=r.role,
content=r.content,
latency_ms=r.latency_ms,
created_at=r.created_at,
)
for r in self.session.exec(statement).all()
]

View File

@ -13,7 +13,19 @@ from agenteval.storage.db import get_session, init_db
from agenteval.storage.repository import RunRepository
from agenteval.version import get_build_info, get_version
from agenteval.web.deps import require_api_key
from agenteval.web.routers import auth, campaigns, files, model_configs, proxy, reports, runs, scenarios, stats, targets
from agenteval.web.routers import (
auth,
campaigns,
exploration,
files,
model_configs,
proxy,
reports,
runs,
scenarios,
stats,
targets,
)
from agenteval.web.websocket import ws_manager
@ -74,6 +86,7 @@ app.include_router(targets.router, prefix="/api/targets", tags=["targets"], depe
app.include_router(scenarios.router, prefix="/api/scenarios", tags=["scenarios"], dependencies=_api_deps)
app.include_router(runs.router, prefix="/api/runs", tags=["runs"], dependencies=_api_deps)
app.include_router(campaigns.router, prefix="/api/campaigns", tags=["campaigns"], dependencies=_api_deps)
app.include_router(exploration.router, prefix="/api/exploration", tags=["exploration"], dependencies=_api_deps)
app.include_router(reports.router, prefix="/api/reports", tags=["reports"], dependencies=_api_deps)
app.include_router(stats.router, prefix="/api/stats", tags=["stats"], dependencies=_api_deps)
app.include_router(files.router, prefix="/api/files", tags=["files"], dependencies=_api_deps)

View File

@ -26,7 +26,7 @@ from agenteval.evaluation.report import (
summarize_campaign_progress,
)
from agenteval.evaluation.report_render import render_campaign_markdown
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, ExplorationBudgetConfig, ExplorationSeeds
from agenteval.storage.db import iso_utc, utc_now
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import (
@ -49,6 +49,8 @@ class CreateCampaignRequest(BaseModel):
time_scale: float = Field(default=1.0, gt=0)
plan: list[CampaignPlanEntry] = Field(min_length=1)
analysis_model_config_id: str | None = None
exploration_seeds: ExplorationSeeds | None = None
exploration_budget: ExplorationBudgetConfig | None = None
@router.get("")
@ -84,6 +86,10 @@ async def create_campaign(
if not ModelConfigRepository(session).get(request.analysis_model_config_id):
raise HTTPException(status_code=400, detail="analysis model config not found")
seeds = request.exploration_seeds
if seeds is not None and not seeds.personas and not seeds.goals:
seeds = None # 种子留空 = 该活动不参与探索
campaign = Campaign(
name=request.name,
target_id=request.target_id,
@ -91,6 +97,8 @@ async def create_campaign(
time_scale=request.time_scale,
plan=request.plan,
analysis_model_config_id=request.analysis_model_config_id,
exploration_seeds=seeds,
exploration_budget=request.exploration_budget,
)
repo = CampaignRepository(session)
campaign = repo.create(campaign)

View File

@ -0,0 +1,293 @@
"""API routes for exploratory evaluation sessions (探索式评测, v0.9).
The virtual user (OpenClaw) drives these sessions through plain HTTP: create a
running session against a campaign, converse with the target through its real
channel, then close with a structured self-reported experience record.
Guardrails are a platform ledger enforced here on every call, never trusted
to client self-discipline. Budget overruns and state violations are rejected
with 409 plus a readable reason, so the rejection itself is feedback to the
resident agent.
"""
from datetime import datetime
from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlmodel import Session
from agenteval.channels.factory import ChannelFactory
from agenteval.config import get_settings
from agenteval.evaluation.report import generate_campaign_report
from agenteval.exploration.models import (
ExplorationBudget,
ExplorationMessage,
ExplorationSession,
ExplorationSessionStatus,
ExplorationTrigger,
normalize_experience,
resolve_budget,
)
from agenteval.models import Campaign, CampaignStatus, EvalRun
from agenteval.storage.db import as_utc, iso_utc, utc_now
from agenteval.storage.repository import (
CampaignRepository,
ExplorationMessageRepository,
ExplorationSessionRepository,
RunRepository,
ScenarioRepository,
TargetRepository,
)
from agenteval.web.deps import get_db
router = APIRouter()
class CreateSessionRequest(BaseModel):
campaign_id: str
persona: dict[str, Any]
goal: str = Field(min_length=1)
seed_ref: dict[str, Any] | None = None
triggered_by: ExplorationTrigger = ExplorationTrigger.AUTO
class SendMessageRequest(BaseModel):
content: str = Field(min_length=1)
class CloseSessionRequest(BaseModel):
experience: dict[str, Any]
def _check_creation_guardrails(
campaign: Campaign,
triggered_by: ExplorationTrigger,
budget: ExplorationBudget,
repo: ExplorationSessionRepository,
) -> None:
if campaign.status != CampaignStatus.RUNNING:
raise HTTPException(status_code=409, detail="活动不在进行中,无法创建探索会话")
if campaign.time_scale != 1 and triggered_by != ExplorationTrigger.MANUAL:
raise HTTPException(
status_code=409,
detail="加速调试线仅允许手动创建探索会话(时间压缩与拟真相冲突)",
)
sessions = repo.list_by_campaign(campaign.id)
if len(sessions) >= budget.max_sessions:
raise HTTPException(
status_code=409,
detail=f"探索会话数超出预算:本活动窗口最多 {budget.max_sessions} 个会话",
)
if sessions:
latest = max(s.created_at for s in sessions if s.created_at)
elapsed = (utc_now() - as_utc(latest)).total_seconds()
if elapsed < budget.min_interval_seconds:
wait_minutes = budget.min_interval_seconds // 60
raise HTTPException(
status_code=409,
detail=f"相邻探索会话间隔不足:最小间隔 {wait_minutes} 分钟,请稍后再试",
)
def _new_runs_since(runs: list[EvalRun], watermark: datetime | None) -> list[EvalRun]:
fresh = []
for run in runs:
if run.completed_at is None:
continue
if watermark is not None and as_utc(run.completed_at) <= watermark:
continue
fresh.append(run)
return fresh
@router.get("/patrol")
async def patrol(session: Session = Depends(get_db)) -> dict:
"""Stateless patrol for the resident agent.
Reports every running production-line (time_scale == 1) campaign that
participates in exploration (has a seed set), with new results since the
last watermark and the remaining exploration budget. Advances each
patrolled campaign's watermark after building the response, so the next
call only reports increments.
"""
campaign_repo = CampaignRepository(session)
run_repo = RunRepository(session)
exploration_repo = ExplorationSessionRepository(session)
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
target_names = {t.id: t.name for t in TargetRepository(session).list_all()}
patrolled_at = utc_now()
entries: list[dict[str, Any]] = []
patrolled_campaigns: list[Campaign] = []
for campaign in campaign_repo.list_all():
if campaign.status != CampaignStatus.RUNNING:
continue
if campaign.time_scale != 1:
continue
if campaign.exploration_seeds is None:
continue
watermark = as_utc(campaign.last_patrolled_at) if campaign.last_patrolled_at else None
fresh = _new_runs_since(run_repo.list_by_campaign(campaign.id), watermark)
new_results = None
if fresh:
report = generate_campaign_report(campaign, fresh, scenario_names=scenario_names)
new_results = {
"summary": report["summary"],
"capability_summary": report["capability_summary"],
}
budget = resolve_budget(campaign)
sessions = exploration_repo.list_by_campaign(campaign.id)
seconds_since_last_session = None
if sessions:
latest = max(as_utc(s.created_at) for s in sessions if s.created_at)
seconds_since_last_session = int((patrolled_at - latest).total_seconds())
entries.append(
{
"campaign_id": campaign.id,
"campaign_name": campaign.name,
"target_id": campaign.target_id,
"target_name": target_names.get(campaign.target_id),
"last_patrolled_at": iso_utc(campaign.last_patrolled_at),
"new_results": new_results,
"budget": {
"max_sessions": budget.max_sessions,
"sessions_used": len(sessions),
"remaining_sessions": max(0, budget.max_sessions - len(sessions)),
"max_turns": budget.max_turns,
"min_interval_seconds": budget.min_interval_seconds,
"seconds_since_last_session": seconds_since_last_session,
},
}
)
patrolled_campaigns.append(campaign)
# 水位取构建响应之后的时刻:查询与持久化之间完成的结果不会在下次重复上报。
watermark_at = utc_now()
for campaign in patrolled_campaigns:
campaign.last_patrolled_at = watermark_at
campaign_repo.update(campaign)
return {"patrolled_at": iso_utc(watermark_at), "campaigns": entries}
@router.post("/sessions")
async def create_session(
request: CreateSessionRequest,
session: Session = Depends(get_db),
) -> dict:
campaign = CampaignRepository(session).get(request.campaign_id)
if not campaign:
raise HTTPException(status_code=404, detail="campaign not found")
if not TargetRepository(session).get(campaign.target_id):
raise HTTPException(status_code=404, detail="campaign target not found")
budget = resolve_budget(campaign)
repo = ExplorationSessionRepository(session)
_check_creation_guardrails(campaign, request.triggered_by, budget, repo)
session_obj = ExplorationSession(
campaign_id=campaign.id,
target_id=campaign.target_id,
persona=request.persona,
goal=request.goal,
seed_ref=request.seed_ref,
triggered_by=request.triggered_by,
)
return repo.create(session_obj).model_dump(mode="json")
@router.post("/sessions/{session_id}/messages")
async def send_session_message(
session_id: str,
request: SendMessageRequest,
session: Session = Depends(get_db),
) -> dict:
repo = ExplorationSessionRepository(session)
session_obj = repo.get(session_id)
if not session_obj:
raise HTTPException(status_code=404, detail="exploration session not found")
if session_obj.status != ExplorationSessionStatus.RUNNING:
raise HTTPException(status_code=409, detail="探索会话不在进行中,拒收消息")
campaign = CampaignRepository(session).get(session_obj.campaign_id)
budget = resolve_budget(campaign) if campaign else ExplorationBudget()
if session_obj.turn_count >= budget.max_turns:
raise HTTPException(
status_code=409,
detail=f"会话轮数超出预算:单会话最多 {budget.max_turns}",
)
target = TargetRepository(session).get(session_obj.target_id)
if not target:
raise HTTPException(status_code=404, detail="session target not found")
channel = ChannelFactory.create(target)
sent_at = utc_now()
try:
send_result = await channel.send(request.content)
except Exception as exc: # channel adapters raise transport-specific errors
raise HTTPException(status_code=502, detail=f"评测对象通道发送失败: {exc}")
if not send_result.ok:
raise HTTPException(
status_code=502,
detail=f"评测对象通道发送失败: {send_result.error}",
)
# 消息已送达即消耗一轮预算(平台账本):先落用户消息,再等回复。
message_repo = ExplorationMessageRepository(session)
round_index = session_obj.turn_count + 1
message_repo.save_message(
ExplorationMessage(
session_id=session_obj.id, round_index=round_index, role="user", content=request.content, created_at=sent_at
)
)
session_obj.turn_count = round_index
repo.update(session_obj)
try:
reply = await channel.poll_reply(
send_result.question_msg_id,
timeout=get_settings().poll_reply_timeout,
)
except Exception as exc:
raise HTTPException(status_code=502, detail=f"等待评测对象回复失败: {exc}")
if reply is None:
raise HTTPException(status_code=502, detail="等待评测对象回复超时")
received_at = utc_now()
latency_ms = int((received_at - sent_at).total_seconds() * 1000)
reply_text = str(reply.content)
message_repo.save_message(
ExplorationMessage(
session_id=session_obj.id,
round_index=round_index,
role="assistant",
content=reply_text,
latency_ms=latency_ms,
created_at=received_at,
)
)
return {"reply": reply_text, "latency_ms": latency_ms, "turn_count": round_index}
@router.post("/sessions/{session_id}/close")
async def close_session(
session_id: str,
request: CloseSessionRequest,
session: Session = Depends(get_db),
) -> dict:
repo = ExplorationSessionRepository(session)
session_obj = repo.get(session_id)
if not session_obj:
raise HTTPException(status_code=404, detail="exploration session not found")
if session_obj.status != ExplorationSessionStatus.RUNNING:
raise HTTPException(status_code=409, detail="探索会话不在进行中,无法关闭")
session_obj.experience = normalize_experience(request.experience)
session_obj.status = ExplorationSessionStatus.COMPLETED
session_obj.closed_at = utc_now()
return repo.update(session_obj).model_dump(mode="json")

View File

@ -17,3 +17,27 @@
- v1 的活动计划是静态的OpenClaw 至多在启动时生成一次,执行期不参与;因此 v1 报告反映的是"预设编排下的周期表现",尚不含 AI 依结果调整的闭环。
- v2 把 OpenClaw 放进调度热回路,其可靠性边界是"决策点可用"(非全程存活);决策点不可用时该时段应可跳过/重试而不拖垮整个活动。
- 活动是**单对象**聚合;多对象对比通过并行活动 + 跨活动对比实现,不在活动内部横向编排(与 ADR-0001「同场景同版本才可比」一致
---
## v2 修订2026-08-03从"重规划"到"虚拟用户",双轨结构
原 v2 设想是 OpenClaw 在决策点重排计划条目。经 grilling 重新定位OpenClaw 的价值不是给固定场景排课表,而是**当虚拟用户本身**——模拟复杂拟真用户行为去打被评对象,产出问题与建议。据此修订为双轨结构:
1. **周期活动内嵌探索(先行)**现有活动骨架不变静态计划照常执行OpenClaw 常驻代理按巡检结果在窗口内派发探索会话OpenClaw 停摆只暂停探索,固定计划照常完成。
2. **探索活动(后续里程碑)**独立活动类型无预设计划OpenClaw 全权接管评测任务。
两轨共用底座探索会话对象、种子配置、体验记录、judge 复核、平台护栏。
**关键决策**(完整清单见 grilling 记录):
- **平台耐久底座 + OpenClaw 常驻代理循环**(原"OpenClaw 全程自主驱动"仍被否,但修正为:常驻的是代理进程/循环而非会话;耐久性归平台,自主性归代理)。可靠性边界沿用原文:"决策点可用",不可用时跳过不拖垮活动。
- **探索会话是独立实体**,不并入 Run——Run 绑定场景与规则判定,探索会话两者皆无,合并会污染 ADR-0001/0002 的可比性与通过率口径。
- **判定 = 体验判定为主 + judge 抽样复核**:意图-结果闭环只有行动者自己能给judge 补客观质量维度,两条证据链在分析层汇合。
- **种子集(种子人设 × 种子目标)是探索式评测的可比性单位**,活动级配置、与 `plan` 同构;代理可在预算内有限衍生。
- **巡检载体**:全局一个 OpenClaw heartbeat/cron 常驻作业 + 平台巡检 APIOpenClaw 调度持久化在 SQLite、重启不丢、连续失败自动禁用
- **护栏平台硬执行**:超预算 API 直接拒绝(默认 ≤8 会话/窗口、≤12 轮/会话、≥30min 间隔)。
- **档位**正式线time_scale == 1自动加速线仅手动——时间压缩与拟真本质冲突。
- **产出接入**:报告新增"探索发现"维度 + 喂 v0.7 分析 + Markdown 导出;周期对比暂不扩展,待口径稳定。
词汇表更新见 `CONTEXT.md`「探索式评测」章节。

View File

@ -313,6 +313,17 @@ export interface CampaignPlanEntry {
count: number
}
export interface ExplorationSeeds {
personas: string[]
goals: string[]
}
export interface ExplorationBudgetConfig {
max_sessions?: number | null
max_turns?: number | null
min_interval_seconds?: number | null
}
export interface CampaignProgress {
current_offset_seconds: number
spawned_runs: number
@ -337,6 +348,8 @@ export interface Campaign {
completed_at: string | null
summary: Record<string, unknown> | null
analysis_model_config_id: string | null
exploration_seeds: ExplorationSeeds | null
exploration_budget: ExplorationBudgetConfig | null
progress?: CampaignProgress
}
@ -427,6 +440,8 @@ export interface CreateCampaignPayload {
time_scale: number
plan: CampaignPlanEntry[]
analysis_model_config_id?: string | null
exploration_seeds?: ExplorationSeeds | null
exploration_budget?: ExplorationBudgetConfig | null
}
// ── Period comparison (v0.8) ────────────────────────────────────

View File

@ -83,6 +83,38 @@ function SectionTitle({ children }: { children: ReactNode }) {
return <div style={{ fontWeight: 600, fontSize: 13, margin: '16px 0 12px' }}>{children}</div>
}
function SeedListText({ name, label, placeholder, addLabel }: {
name: string
label: string
placeholder: string
addLabel: string
}) {
return (
<>
<div style={{ marginBottom: 6, fontSize: 12, color: colors.textSecondary }}>{label}</div>
<Form.List name={name}>
{(fields, { add, remove }) => (
<>
{fields.map((field) => (
<div key={field.key} style={{ display: 'flex', gap: 8, marginBottom: 8 }}>
<Form.Item name={field.name} style={{ flex: 1, marginBottom: 0 }}>
<Input placeholder={placeholder} />
</Form.Item>
<div style={{ width: 16, lineHeight: '32px' }}>
<MinusCircleOutlined onClick={() => remove(field.name)} style={{ color: colors.textMuted }} />
</div>
</div>
))}
<Button type="dashed" size="small" onClick={() => add('')} block icon={<PlusOutlined />}>
{addLabel}
</Button>
</>
)}
</Form.List>
</>
)
}
function fmtWindow(seconds: number): string {
if (seconds % 3600 === 0) return `${seconds / 3600}h`
if (seconds % 60 === 0) return `${seconds / 60}m`
@ -181,6 +213,9 @@ export default function CampaignsPage() {
realtime: false, target_value: 60, target_unit: 60,
plan: [{ scenario_id: undefined, offset_hours: 0, count: 1 }],
analysis_model_config_id: null,
seed_personas: [], seed_goals: [],
exploration_max_sessions: undefined, exploration_max_turns: undefined,
exploration_min_interval_minutes: undefined,
})
setCreateOpen(true)
}
@ -193,6 +228,16 @@ export default function CampaignsPage() {
)
setSubmitting(true)
try {
const personas = ((values.seed_personas as string[] | undefined) ?? [])
.map((s) => s?.trim()).filter(Boolean) as string[]
const goals = ((values.seed_goals as string[] | undefined) ?? [])
.map((s) => s?.trim()).filter(Boolean) as string[]
const budget: Record<string, number> = {}
if (values.exploration_max_sessions != null) budget.max_sessions = values.exploration_max_sessions
if (values.exploration_max_turns != null) budget.max_turns = values.exploration_max_turns
if (values.exploration_min_interval_minutes != null) {
budget.min_interval_seconds = values.exploration_min_interval_minutes * 60
}
await campaignsApi.create({
name: values.name,
target_id: values.target_id,
@ -204,6 +249,8 @@ export default function CampaignsPage() {
count: e.count ?? 1,
})),
analysis_model_config_id: (values.analysis_model_config_id as string | null) ?? null,
exploration_seeds: personas.length || goals.length ? { personas, goals } : null,
exploration_budget: Object.keys(budget).length ? budget : null,
})
message.success('评估活动已创建并开始调度')
setCreateOpen(false)
@ -897,6 +944,35 @@ export default function CampaignsPage() {
</>
)}
</Form.List>
<SectionTitle>
{' '}
<Tooltip title="AI 助手以种子人设 × 种子目标自主与被评对象对话;种子留空则该活动不参与探索">
<QuestionCircleOutlined style={{ color: colors.textMuted }} />
</Tooltip>
</SectionTitle>
<Row gutter={16}>
<Col span={12}>
<SeedListText name="seed_personas" label="种子人设" placeholder="如:急性子的缴费用户" addLabel="添加人设" />
</Col>
<Col span={12}>
<SeedListText name="seed_goals" label="种子目标" placeholder="如:查询本月账单并完成缴费" addLabel="添加目标" />
</Col>
</Row>
<div style={{ marginTop: 12, marginBottom: 6, fontSize: 12, color: colors.textSecondary }}>
使8 / · 12 / · 30
</div>
<div style={{ display: 'flex', gap: 8 }}>
<Form.Item name="exploration_max_sessions" style={{ flex: 1, marginBottom: 0 }}>
<InputNumber min={1} placeholder="默认 8" addonAfter="会话/窗口" style={{ width: '100%' }} />
</Form.Item>
<Form.Item name="exploration_max_turns" style={{ flex: 1, marginBottom: 0 }}>
<InputNumber min={1} placeholder="默认 12" addonAfter="轮/会话" style={{ width: '100%' }} />
</Form.Item>
<Form.Item name="exploration_min_interval_minutes" style={{ flex: 1, marginBottom: 0 }}>
<InputNumber min={1} placeholder="默认 30" addonAfter="分钟间隔" style={{ width: '100%' }} />
</Form.Item>
</div>
</Col>
</Row>
</Form>

View File

@ -0,0 +1,69 @@
"""add exploration_sessions and exploration_messages tables
Revision ID: 0e4a7c91d2b3
Revises: f2a9b7c34d18
Create Date: 2026-08-03
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
revision: str = "0e4a7c91d2b3"
down_revision: Union[str, Sequence[str], None] = "f2a9b7c34d18"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"exploration_sessions",
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("campaign_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("target_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("persona", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("goal", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("seed_ref", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("status", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("triggered_by", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("experience", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("judge_review", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("turn_count", sa.Integer(), nullable=False),
sa.Column("error", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("created_at", sa.DateTime(), nullable=True),
sa.Column("closed_at", sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint("id"),
sa.ForeignKeyConstraint(["campaign_id"], ["campaigns.id"]),
sa.ForeignKeyConstraint(["target_id"], ["eval_targets.id"]),
)
op.create_index(
"ix_exploration_sessions_campaign_id",
"exploration_sessions",
["campaign_id"],
)
op.create_table(
"exploration_messages",
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("session_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("round_index", sa.Integer(), nullable=False),
sa.Column("role", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("content", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("latency_ms", sa.Integer(), nullable=True),
sa.Column("created_at", sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint("id"),
sa.ForeignKeyConstraint(["session_id"], ["exploration_sessions.id"]),
)
op.create_index(
"ix_exploration_messages_session_id",
"exploration_messages",
["session_id"],
)
def downgrade() -> None:
op.drop_index("ix_exploration_messages_session_id")
op.drop_table("exploration_messages")
op.drop_index("ix_exploration_sessions_campaign_id")
op.drop_table("exploration_sessions")

View File

@ -0,0 +1,29 @@
"""add exploration seeds/budget config to campaigns
Revision ID: b3c7d9e1f5a2
Revises: 0e4a7c91d2b3
Create Date: 2026-08-03
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
revision: str = "b3c7d9e1f5a2"
down_revision: Union[str, Sequence[str], None] = "0e4a7c91d2b3"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("campaigns") as batch_op:
batch_op.add_column(sa.Column("exploration_seeds", sqlmodel.sql.sqltypes.AutoString(), nullable=True))
batch_op.add_column(sa.Column("exploration_budget", sqlmodel.sql.sqltypes.AutoString(), nullable=True))
def downgrade() -> None:
with op.batch_alter_table("campaigns") as batch_op:
batch_op.drop_column("exploration_budget")
batch_op.drop_column("exploration_seeds")

View File

@ -0,0 +1,26 @@
"""add campaigns.last_patrolled_at watermark
Revision ID: c5d8f0a2e4b7
Revises: b3c7d9e1f5a2
Create Date: 2026-08-03
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "c5d8f0a2e4b7"
down_revision: Union[str, Sequence[str], None] = "b3c7d9e1f5a2"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("campaigns") as batch_op:
batch_op.add_column(sa.Column("last_patrolled_at", sa.DateTime(), nullable=True))
def downgrade() -> None:
with op.batch_alter_table("campaigns") as batch_op:
batch_op.drop_column("last_patrolled_at")

View File

@ -35,6 +35,8 @@ def db_session(tmp_db_path: Path) -> Session:
EvalResultDB,
EvalRunDB,
EvalTargetDB,
ExplorationMessageDB,
ExplorationSessionDB,
ModelConfigDB,
ScenarioDB,
ScenarioModelBindingDB,
@ -49,6 +51,7 @@ def db_session(tmp_db_path: Path) -> Session:
# Sanity check: verify the scenarios table has all expected columns.
from sqlalchemy import inspect as sa_inspect
cols = [c["name"] for c in sa_inspect(engine).get_columns("scenarios")]
assert "llm_config" in cols, f"scenarios table missing llm_config; cols={cols}"

View File

@ -377,3 +377,64 @@ async def test_create_campaign_without_override_stores_null(client, seeded_db):
async def test_create_campaign_invalid_analysis_model_400(client, seeded_db):
resp = await client.post("/api/campaigns", json=_valid_payload(analysis_model_config_id="nope"))
assert resp.status_code == 400
async def test_create_campaign_with_exploration_config(client, seeded_db):
payload = _valid_payload(
exploration_seeds={"personas": ["急性子用户", "谨慎的老年用户"], "goals": ["查询账单并缴费", "修改收货地址"]},
exploration_budget={"max_sessions": 4, "max_turns": 6, "min_interval_seconds": 3600},
)
resp = await client.post("/api/campaigns", json=payload)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["exploration_seeds"] == {
"personas": ["急性子用户", "谨慎的老年用户"],
"goals": ["查询账单并缴费", "修改收货地址"],
}
assert body["exploration_budget"] == {"max_sessions": 4, "max_turns": 6, "min_interval_seconds": 3600}
got = (await client.get(f"/api/campaigns/{body['id']}")).json()
assert got["exploration_seeds"]["personas"] == ["急性子用户", "谨慎的老年用户"]
assert got["exploration_budget"]["max_turns"] == 6
async def test_create_campaign_without_exploration_config(client, seeded_db):
body = (await client.post("/api/campaigns", json=_valid_payload())).json()
assert body["exploration_seeds"] is None
assert body["exploration_budget"] is None
async def test_empty_seeds_campaign_opts_out_of_exploration(client, seeded_db):
payload = _valid_payload(exploration_seeds={"personas": [], "goals": []})
body = (await client.post("/api/campaigns", json=payload)).json()
assert body["exploration_seeds"] is None
def test_exploration_config_migration_on_existing_db(tmp_path, monkeypatch):
"""The two campaign config columns apply on a DB at the previous head."""
from pathlib import Path
from agenteval.storage import db as db_module
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
from sqlmodel import SQLModel
database_url = f"sqlite:///{tmp_path / 'campaign_config.db'}"
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
SQLModel.metadata.create_all(create_engine(database_url))
with create_engine(database_url).begin() as connection:
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
command.stamp(config, "0e4a7c91d2b3")
command.upgrade(config, "head")
cols = {c["name"] for c in inspect(create_engine(database_url)).get_columns("campaigns")}
assert {"exploration_seeds", "exploration_budget"} <= cols

View File

@ -0,0 +1,391 @@
"""Integration tests for exploration session lifecycle + platform guardrails (v0.9 票据 01).
Uses ``httpx.AsyncClient`` with ``app=`` to drive the FastAPI app in-process.
Channel I/O is stubbed with MockChannel; tests cover the full lifecycle
(create message close), the three budget guardrails (409), trigger-source
gating per line tier, and experience-record normalization.
"""
from datetime import timedelta
import pytest
from agenteval.models import Campaign, ChannelType, EvalTarget, PlatformType, TargetStatus
from agenteval.storage.db import ExplorationSessionDB, utc_now
from agenteval.storage.repository import CampaignRepository, ExplorationSessionRepository, TargetRepository
from agenteval.web.app import app
from httpx import ASGITransport, AsyncClient
def _make_campaign(campaign_id: str, *, time_scale: float = 1.0, status: str = "running") -> Campaign:
return Campaign(
id=campaign_id,
name=f"campaign-{campaign_id}",
target_id="t-1",
window_seconds=86400,
time_scale=time_scale,
plan=[{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
status=status,
started_at=utc_now(),
)
@pytest.fixture()
def seeded_db(db_session, monkeypatch):
"""Patch get_session/get_db to the test session and seed target + campaign."""
from agenteval.storage import db as db_module
from agenteval.storage import repository as repo_module
from agenteval.web import app as app_module
monkeypatch.setattr(app_module, "init_db", lambda: None)
def _test_get_session():
return db_session
monkeypatch.setattr(db_module, "get_session", _test_get_session)
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
from agenteval.web.deps import get_db
def _test_get_db():
try:
yield db_session
finally:
pass
app.dependency_overrides[get_db] = _test_get_db
target = EvalTarget(
id="t-1",
name="mock-target",
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API,
channel_config={"base_url": "http://mock", "token": "x"},
status=TargetStatus.ACTIVE,
)
TargetRepository(db_session).create(target)
CampaignRepository(db_session).create(_make_campaign("c-1"))
yield db_session
app.dependency_overrides.clear()
def _stub_channel_factory(monkeypatch, channel) -> None:
"""Point the exploration router's ChannelFactory at a test channel."""
from agenteval.web.routers import exploration as exploration_module
class _StubFactory:
@staticmethod
def create(target):
return channel
monkeypatch.setattr(exploration_module, "ChannelFactory", _StubFactory)
@pytest.fixture()
def mock_channel(monkeypatch):
"""Stub ChannelFactory in the exploration router with a MockChannel."""
from tests.unit.mock_channel import MockChannel
channel = MockChannel(reply_text="您好,请问有什么可以帮您?")
_stub_channel_factory(monkeypatch, channel)
return channel
@pytest.fixture()
async def client():
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
def _session_payload(**overrides) -> dict:
payload = {
"campaign_id": "c-1",
"persona": {"name": "急性子用户", "traits": ["急躁", "目标导向"]},
"goal": "查询本月账单并完成缴费",
"triggered_by": "auto",
}
payload.update(overrides)
return payload
def _rewind_latest_session(db_session, minutes: int = 31) -> None:
"""Move the newest session's created_at back so the interval guardrail passes."""
repo = ExplorationSessionRepository(db_session)
sessions = repo.list_by_campaign("c-1")
latest = max(sessions, key=lambda s: s.created_at)
row = db_session.get(ExplorationSessionDB, latest.id)
row.created_at = utc_now() - timedelta(minutes=minutes)
db_session.add(row)
db_session.commit()
async def _create_session(client, **overrides):
return await client.post("/api/exploration/sessions", json=_session_payload(**overrides))
# ---------------------------------------------------------------- lifecycle
async def test_full_lifecycle_create_message_close(seeded_db, mock_channel, client):
resp = await _create_session(client)
assert resp.status_code == 200, resp.text
session = resp.json()
assert session["status"] == "running"
assert session["campaign_id"] == "c-1"
assert session["target_id"] == "t-1"
assert session["persona"]["name"] == "急性子用户"
session_id = session["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "我要查这个月的账单"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["reply"] == "您好,请问有什么可以帮您?"
assert isinstance(body["latency_ms"], int)
assert body["turn_count"] == 1
repo = ExplorationSessionRepository(seeded_db)
assert repo.get(session_id).turn_count == 1
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close",
json={
"experience": {
"goal_achieved": True,
"blockers": [],
"misled": [],
"emotion": "positive",
"notes": "顺利完成",
}
},
)
assert resp.status_code == 200, resp.text
closed = resp.json()
assert closed["status"] == "completed"
assert closed["experience"]["goal_achieved"] is True
assert closed["experience"]["emotion"] == "positive"
assert closed["closed_at"] is not None
async def test_create_requires_existing_running_campaign(seeded_db, client):
resp = await _create_session(client, campaign_id="nope")
assert resp.status_code == 404
CampaignRepository(seeded_db).create(_make_campaign("c-done", status="completed"))
resp = await _create_session(client, campaign_id="c-done")
assert resp.status_code == 409
assert "进行中" in resp.json()["detail"]
async def test_accelerated_line_accepts_manual_only(seeded_db, mock_channel, client):
CampaignRepository(seeded_db).create(_make_campaign("c-fast", time_scale=24.0))
resp = await _create_session(client, campaign_id="c-fast", triggered_by="auto")
assert resp.status_code == 409
assert "手动" in resp.json()["detail"]
resp = await _create_session(client, campaign_id="c-fast", triggered_by="manual")
assert resp.status_code == 200
async def test_session_budget_guardrail(seeded_db, mock_channel, client):
for _ in range(8):
resp = await _create_session(client)
assert resp.status_code == 200, resp.text
_rewind_latest_session(seeded_db)
resp = await _create_session(client)
assert resp.status_code == 409
assert "预算" in resp.json()["detail"]
async def test_session_interval_guardrail(seeded_db, mock_channel, client):
resp = await _create_session(client)
assert resp.status_code == 200
resp = await _create_session(client)
assert resp.status_code == 409
assert "间隔" in resp.json()["detail"]
_rewind_latest_session(seeded_db)
resp = await _create_session(client)
assert resp.status_code == 200
async def test_campaign_budget_override_limits_sessions(seeded_db, mock_channel, client):
"""活动级预算覆盖生效(票据 02max_sessions=1 时第二个会话被拒。"""
campaign = CampaignRepository(seeded_db).get("c-1")
campaign.exploration_budget = {"max_sessions": 1}
CampaignRepository(seeded_db).update(campaign)
assert (await _create_session(client)).status_code == 200
_rewind_latest_session(seeded_db)
resp = await _create_session(client)
assert resp.status_code == 409
assert "预算" in resp.json()["detail"]
async def test_turn_budget_guardrail(seeded_db, mock_channel, client):
session_id = (await _create_session(client)).json()["id"]
for i in range(12):
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": f"{i + 1} 轮问题"},
)
assert resp.status_code == 200, resp.text
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "第 13 轮问题"},
)
assert resp.status_code == 409
assert "轮数" in resp.json()["detail"]
async def test_message_rejected_when_session_not_running(seeded_db, mock_channel, client):
session_id = (await _create_session(client)).json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close",
json={"experience": {"goal_achieved": False}},
)
assert resp.status_code == 200
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "还在吗?"},
)
assert resp.status_code == 409
assert "进行中" in resp.json()["detail"]
async def test_message_unknown_session_returns_404(seeded_db, mock_channel, client):
resp = await client.post("/api/exploration/sessions/nope/messages", json={"content": "hi"})
assert resp.status_code == 404
async def test_channel_failure_returns_502_without_consuming_turn(seeded_db, monkeypatch, client):
from tests.unit.mock_channel import MockChannel
channel = MockChannel(send_ok=False)
_stub_channel_factory(monkeypatch, channel)
session_id = (await _create_session(client)).json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "你好"},
)
assert resp.status_code == 502
assert ExplorationSessionRepository(seeded_db).get(session_id).turn_count == 0
async def test_poll_timeout_consumes_turn_budget(seeded_db, monkeypatch, client):
"""消息已送达但等不到回复:账本仍计一轮(超时不可绕过轮数预算)。"""
from types import SimpleNamespace
from agenteval.web.routers import exploration as exploration_module
from tests.unit.mock_channel import MockChannel
channel = MockChannel(missing_reply=True)
_stub_channel_factory(monkeypatch, channel)
monkeypatch.setattr(exploration_module, "get_settings", lambda: SimpleNamespace(poll_reply_timeout=0.05))
session_id = (await _create_session(client)).json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/messages",
json={"content": "有人在吗"},
)
assert resp.status_code == 502
assert ExplorationSessionRepository(seeded_db).get(session_id).turn_count == 1
async def test_close_normalizes_experience(seeded_db, mock_channel, client):
session_id = (await _create_session(client)).json()["id"]
resp = await client.post(
f"/api/exploration/sessions/{session_id}/close",
json={
"experience": {
"blockers": [42, {"not": "a string"}],
"misled": "不是列表",
"emotion": "暴怒!!!",
}
},
)
assert resp.status_code == 200, resp.text
experience = resp.json()["experience"]
assert experience["goal_achieved"] is False
assert experience["blockers"] == ["42"]
assert experience["misled"] == []
assert experience["emotion"] == "neutral"
async def test_close_twice_rejected(seeded_db, mock_channel, client):
session_id = (await _create_session(client)).json()["id"]
payload = {"experience": {"goal_achieved": True}}
assert (await client.post(f"/api/exploration/sessions/{session_id}/close", json=payload)).status_code == 200
resp = await client.post(f"/api/exploration/sessions/{session_id}/close", json=payload)
assert resp.status_code == 409
# ---------------------------------------------------------------- migration
def test_exploration_migration_on_existing_db(tmp_path, monkeypatch):
"""Alembic migration applies on an existing DB at the previous head.
Brand-new DBs take the create_all path (exercised by every test above via
the db_session fixture, which creates the new tables from metadata).
"""
from pathlib import Path
from agenteval.storage import db as db_module
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect
database_url = f"sqlite:///{tmp_path / 'exploration.db'}"
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
# 既有库先例:基线迁移假设 create_all 建好的表已存在,先 stamp 基线前状态
from sqlmodel import SQLModel
engine = create_engine(database_url)
SQLModel.metadata.create_all(engine)
engine.dispose()
with create_engine(database_url).begin() as connection:
from sqlalchemy import text
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
command.stamp(config, "f2a9b7c34d18")
command.upgrade(config, "head")
inspector = inspect(create_engine(database_url))
tables = set(inspector.get_table_names())
assert "exploration_sessions" in tables
assert "exploration_messages" in tables
session_cols = {c["name"] for c in inspector.get_columns("exploration_sessions")}
assert {
"campaign_id",
"target_id",
"persona",
"goal",
"seed_ref",
"status",
"triggered_by",
"experience",
"judge_review",
"turn_count",
"closed_at",
} <= session_cols
message_cols = {c["name"] for c in inspector.get_columns("exploration_messages")}
assert {"session_id", "round_index", "role", "content", "latency_ms"} <= message_cols

View File

@ -0,0 +1,203 @@
"""Integration tests for the exploration patrol API (v0.9 票据 03).
One stateless call returns every running production-line campaign that
participates in exploration, the new results since the last patrol watermark
(reusing the campaign report aggregation), and the remaining exploration
budget. The watermark advances after each call so subsequent calls only
report increments.
"""
from datetime import timedelta
import pytest
from agenteval.models import Campaign, EvalRun, RunStatus, RunSummary
from agenteval.storage.db import utc_now
from agenteval.storage.repository import CampaignRepository, RunRepository
from agenteval.web.app import app
from httpx import ASGITransport, AsyncClient
SEEDS = {"personas": ["急性子用户"], "goals": ["查询账单并缴费"]}
def _make_campaign(campaign_id: str, *, time_scale: float = 1.0, status: str = "running", seeds=SEEDS) -> Campaign:
return Campaign(
id=campaign_id,
name=f"campaign-{campaign_id}",
target_id="t-1",
window_seconds=86400,
time_scale=time_scale,
plan=[{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
status=status,
started_at=utc_now() - timedelta(hours=2),
exploration_seeds=seeds,
)
def _seed_run(db_session, run_id: str, campaign_id: str, *, pass_rate: float, completed_at) -> None:
RunRepository(db_session).create(EvalRun(
id=run_id, target_id="t-1", scenario_id="s-1", campaign_id=campaign_id,
status=RunStatus.COMPLETED, started_at=completed_at - timedelta(minutes=5),
completed_at=completed_at,
summary=RunSummary(total_cases=2, pass_rate=pass_rate, avg_latency_ms=120.0),
))
@pytest.fixture()
def seeded_db(db_session, monkeypatch):
from agenteval.models import ChannelType, EvalTarget, PlatformType, TargetStatus
from agenteval.storage import db as db_module
from agenteval.storage import repository as repo_module
from agenteval.storage.repository import TargetRepository
from agenteval.web import app as app_module
monkeypatch.setattr(app_module, "init_db", lambda: None)
def _test_get_session():
return db_session
monkeypatch.setattr(db_module, "get_session", _test_get_session)
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
from agenteval.web.deps import get_db
def _test_get_db():
try:
yield db_session
finally:
pass
app.dependency_overrides[get_db] = _test_get_db
TargetRepository(db_session).create(EvalTarget(
id="t-1", name="mock-target",
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API,
channel_config={"base_url": "http://mock", "token": "x"},
status=TargetStatus.ACTIVE,
))
repo = CampaignRepository(db_session)
repo.create(_make_campaign("c-prod")) # 正式线,参与探索
repo.create(_make_campaign("c-fast", time_scale=24.0)) # 加速线 → 不巡检
repo.create(_make_campaign("c-noseed", seeds=None)) # 无种子集 → 不巡检
repo.create(_make_campaign("c-done", status="completed")) # 终态 → 不巡检
_seed_run(db_session, "r-1", "c-prod", pass_rate=1.0, completed_at=utc_now() - timedelta(hours=1))
_seed_run(db_session, "r-2", "c-prod", pass_rate=0.5, completed_at=utc_now() - timedelta(minutes=30))
yield db_session
app.dependency_overrides.clear()
@pytest.fixture()
async def client():
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
async def _patrol(client) -> dict:
resp = await client.get("/api/exploration/patrol")
assert resp.status_code == 200, resp.text
return resp.json()
async def test_patrol_filters_to_running_production_seeded_campaigns(client, seeded_db):
body = await _patrol(client)
ids = [c["campaign_id"] for c in body["campaigns"]]
assert ids == ["c-prod"]
async def test_patrol_entry_content(client, seeded_db):
body = await _patrol(client)
entry = body["campaigns"][0]
assert entry["campaign_name"] == "campaign-c-prod"
assert entry["target_id"] == "t-1"
assert entry["target_name"] == "mock-target"
assert entry["last_patrolled_at"] is None # 首次巡检无水位
new_results = entry["new_results"]
assert new_results is not None
assert new_results["summary"]["total_runs"] == 2
assert new_results["summary"]["overall_pass_rate"] == 0.75
assert new_results["capability_summary"][0]["scenario_id"] == "s-1"
budget = entry["budget"]
assert budget["max_sessions"] == 8
assert budget["sessions_used"] == 0
assert budget["remaining_sessions"] == 8
assert budget["max_turns"] == 12
assert budget["min_interval_seconds"] == 30 * 60
assert budget["seconds_since_last_session"] is None
async def test_patrol_watermark_advances_and_reports_increments(client, seeded_db):
await _patrol(client)
campaign = CampaignRepository(seeded_db).get("c-prod")
assert campaign.last_patrolled_at is not None
# 第二次巡检:无新完成的子运行 → 增量为空
body = await _patrol(client)
entry = body["campaigns"][0]
assert entry["new_results"] is None
# 水位之后新完成一个子运行 → 第三次巡检只报这一个
_seed_run(seeded_db, "r-3", "c-prod", pass_rate=0.0, completed_at=utc_now())
body = await _patrol(client)
entry = body["campaigns"][0]
assert entry["new_results"]["summary"]["total_runs"] == 1
assert entry["new_results"]["summary"]["overall_pass_rate"] == 0.0
async def test_patrol_budget_reflects_existing_sessions(client, seeded_db):
from agenteval.exploration.models import ExplorationSession
from agenteval.storage.repository import ExplorationSessionRepository
ExplorationSessionRepository(seeded_db).create(ExplorationSession(
campaign_id="c-prod", target_id="t-1", goal="查询账单", persona={"name": "x"},
))
body = await _patrol(client)
budget = body["campaigns"][0]["budget"]
assert budget["sessions_used"] == 1
assert budget["remaining_sessions"] == 7
assert budget["seconds_since_last_session"] is not None
async def test_patrol_budget_honours_campaign_override(client, seeded_db):
campaign = CampaignRepository(seeded_db).get("c-prod")
campaign.exploration_budget = {"max_sessions": 3}
CampaignRepository(seeded_db).update(campaign)
body = await _patrol(client)
budget = body["campaigns"][0]["budget"]
assert budget["max_sessions"] == 3
assert budget["remaining_sessions"] == 3
async def test_patrol_migration_column_on_existing_db(tmp_path, monkeypatch):
"""last_patrolled_at applies on a DB at the previous head."""
from pathlib import Path
from agenteval.storage import db as db_module
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
from sqlmodel import SQLModel
database_url = f"sqlite:///{tmp_path / 'patrol.db'}"
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
SQLModel.metadata.create_all(create_engine(database_url))
with create_engine(database_url).begin() as connection:
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
command.stamp(config, "b3c7d9e1f5a2")
command.upgrade(config, "head")
cols = {c["name"] for c in inspect(create_engine(database_url)).get_columns("campaigns")}
assert "last_patrolled_at" in cols

View File

@ -0,0 +1,38 @@
"""Unit tests for exploration budget resolution (platform defaults + campaign override)."""
from agenteval.exploration.models import resolve_budget
from agenteval.models import Campaign
def _campaign(**overrides) -> Campaign:
base = {
"name": "c",
"target_id": "t-1",
"window_seconds": 86400,
"plan": [{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
}
base.update(overrides)
return Campaign(**base)
def test_platform_defaults_when_no_override():
budget = resolve_budget(_campaign())
assert budget.max_sessions == 8
assert budget.max_turns == 12
assert budget.min_interval_seconds == 30 * 60
def test_partial_override_falls_back_to_defaults():
campaign = _campaign(exploration_budget={"max_sessions": 3})
budget = resolve_budget(campaign)
assert budget.max_sessions == 3
assert budget.max_turns == 12
assert budget.min_interval_seconds == 30 * 60
def test_full_override():
campaign = _campaign(
exploration_budget={"max_sessions": 2, "max_turns": 5, "min_interval_seconds": 600}
)
budget = resolve_budget(campaign)
assert (budget.max_sessions, budget.max_turns, budget.min_interval_seconds) == (2, 5, 600)