Compare commits
5 Commits
2d7679a6bb
...
3e485bfbe6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3e485bfbe6 | ||
|
|
6340ec503c | ||
|
|
53afb9b5d1 | ||
|
|
a351f65550 | ||
|
|
c35159ffcb |
13
.scratch/v0.9/issues/01-session-lifecycle-and-guardrails.md
Normal file
13
.scratch/v0.9/issues/01-session-lifecycle-and-guardrails.md
Normal file
@ -0,0 +1,13 @@
|
||||
# 01 — 探索会话生命周期 + 平台硬护栏
|
||||
|
||||
**What to build:** 从 API 视角完整走通一段探索会话:指定活动、人设与目标创建 running 会话 → 以虚拟用户身份发消息并收到被评对象回复(经目标通道转发,双方轮次持久化、记录延迟)→ 提交结构化体验记录(达成、卡点、被误导、情绪)关闭会话。护栏是平台账本:超窗口会话数、超单会话轮数、相邻会话间隔不足,一律 409 并附原因;会话非 running 状态拒收消息。探索会话是独立实体,不并入评测运行(不污染通过率口径)。
|
||||
|
||||
**Blocked by:** None — can start immediately
|
||||
|
||||
**Status:** completed
|
||||
|
||||
- [x] 探索会话与会话轮次独立建表,迁移可在全新库与既有库上应用
|
||||
- [x] 创建/发消息/关闭三个端点全链路可用,X-API-Key 鉴权
|
||||
- [x] 三类预算超限各返回 409 + 可读原因;平台默认值生效(≤8 会话/窗口、≤12 轮/会话、≥30min 间隔)
|
||||
- [x] 体验记录结构校验,非法值归一(沿 v0.7 白名单经验)
|
||||
- [x] 集成测试覆盖完整生命周期与各拒绝分支(先例:活动 API 集成测试)
|
||||
13
.scratch/v0.9/issues/02-campaign-seeds-and-budget.md
Normal file
13
.scratch/v0.9/issues/02-campaign-seeds-and-budget.md
Normal file
@ -0,0 +1,13 @@
|
||||
# 02 — 活动级种子集与探索预算配置
|
||||
|
||||
**What to build:** 创建活动时可为该活动配置种子集(种子人设数组 + 种子目标数组)与探索预算(最大会话数 / 单会话最大轮数 / 会话最小间隔,空则用平台默认)。配置与活动计划同构入库,可查询;前端活动创建表单提供种子与预算输入,种子留空即该活动不参与探索。种子集是探索式评测的可比性单位,随活动持久化。
|
||||
|
||||
**Blocked by:** None — can start immediately
|
||||
|
||||
**Status:** completed
|
||||
|
||||
- [x] 活动实体新增种子集与预算两个 JSON 配置字段,迁移可在全新库与既有库上应用
|
||||
- [x] 活动创建/查询 API 接受并返回配置;预算空值回落平台默认
|
||||
- [x] 创建表单可录入种子人设、种子目标与预算覆盖
|
||||
- [x] 种子留空的活动创建成功且标记为不参与探索
|
||||
- [x] 集成测试覆盖配置存取与默认回落
|
||||
12
.scratch/v0.9/issues/03-patrol-api.md
Normal file
12
.scratch/v0.9/issues/03-patrol-api.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 03 — 巡检 API
|
||||
|
||||
**What to build:** 常驻代理一次调用即可无状态巡检:返回所有进行中的正式线活动(time_scale == 1)清单,每项含活动标识与对象、自上次巡检以来的新结果摘要(复用现有活动报告聚合口径,不重算新指标)、探索预算余量(剩余会话数等)。响应生成后推进各活动的巡检水位,使下次调用只报增量。加速线活动不出现在巡检结果中。
|
||||
|
||||
**Blocked by:** 01(预算余量需会话计数)、02(活动预算字段)
|
||||
|
||||
**Status:** completed
|
||||
|
||||
- [x] 巡检端点返回进行中正式线活动 + 新结果摘要 + 预算余量
|
||||
- [x] 巡检水位推进:连续两次调用,第二次只含增量
|
||||
- [x] 加速线与终态活动不出现
|
||||
- [x] 集成测试覆盖内容正确性、水位推进与过滤规则
|
||||
12
.scratch/v0.9/issues/04-judge-review.md
Normal file
12
.scratch/v0.9/issues/04-judge-review.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 04 — judge 抽样复核
|
||||
|
||||
**What to build:** 会话关闭后,平台后台对该会话对话抽样(默认 ≤3 段,控 token),经 judge 岗位模型独立复核,产出质量维度结论(态度、专业性、幻觉等)落入会话记录。复核是异步后台执行:失败落错误不阻塞会话状态,也不影响体验记录这条第一手证据线。LLM 调用走可注入客户端接缝,测试用假客户端覆盖。
|
||||
|
||||
**Blocked by:** 01(会话关闭事件与对话数据)
|
||||
|
||||
**Status:** ready-for-agent
|
||||
|
||||
- [ ] 会话关闭后自动触发抽样复核,结果结构化落库
|
||||
- [ ] 抽样上限生效;无模型配置时静默跳过
|
||||
- [ ] 复核失败落错误,会话仍为 completed
|
||||
- [ ] 假客户端测试覆盖编排、解析失败、无模型分支(先例:v0.7 分析测试)
|
||||
12
.scratch/v0.9/issues/05-report-analysis-export.md
Normal file
12
.scratch/v0.9/issues/05-report-analysis-export.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 05 — 探索发现 → 报告 / 分析 / 导出
|
||||
|
||||
**What to build:** 探索数据汇入既有报告链的三个出口。活动报告新增"探索发现"维度:会话数、目标达成率、问题清单(来自体验记录聚合;judge 复核结论若已就位一并纳入)。v0.7 活动分析的输入追加探索摘要(问题清单 + 达成统计,非全量对话),让分析模型合成两条证据线。Markdown 导出在存在探索数据时追加「探索发现」附录(顺序在智能分析与周期对比之后),无探索数据时导出与旧版完全一致。周期对比口径不变。
|
||||
|
||||
**Blocked by:** 01(聚合主料是体验记录);04 若已就位则复核结论一并纳入,未就位不阻塞
|
||||
|
||||
**Status:** ready-for-agent
|
||||
|
||||
- [ ] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
|
||||
- [ ] 分析输入含探索摘要且不含全量对话
|
||||
- [ ] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
|
||||
- [ ] 纯函数渲染与聚合单测 + 导出集成测试(先例:report_render 单测、活动导出测试)
|
||||
12
.scratch/v0.9/issues/06-drawer-exploration-section.md
Normal file
12
.scratch/v0.9/issues/06-drawer-exploration-section.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 06 — 报告抽屉「探索发现」区块 + 会话下钻
|
||||
|
||||
**What to build:** 活动报告抽屉新增「探索发现」区块(模式沿 v0.7 智能分析区块):会话数、目标达成率、问题清单、会话列表状态。每个会话可点开查看完整对话与体验记录——证据下钻是探索数据的命脉,"用户在这里被绕晕"必须能看到原文。无探索数据时区块不出现。不建新顶级页面。
|
||||
|
||||
**Blocked by:** 05(报告聚合与查询端点就绪)
|
||||
|
||||
**Status:** ready-for-agent
|
||||
|
||||
- [ ] 报告抽屉呈现探索发现区块,数据与后端口径一致
|
||||
- [ ] 会话详情展示完整对话(角色、内容、延迟)与体验记录
|
||||
- [ ] 无探索数据时区块隐藏
|
||||
- [ ] tsc 通过;人工在浏览器验证展开/下钻交互
|
||||
12
.scratch/v0.9/issues/07-finalize-and-line-gates.md
Normal file
12
.scratch/v0.9/issues/07-finalize-and-line-gates.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 07 — 活动 finalize 会话结算 + 档位校验
|
||||
|
||||
**What to build:** 活动生命周期与探索会话的收口规则。窗口结束进入终态结算时,仍 running 的探索会话转 expired,此后拒收消息——不留悬挂会话。档位校验完整化:正式线接受自动与手动触发的会话;加速调试线仅接受手动触发,自动来源一律拒绝(时间压缩与拟真冲突,加速线不参与自动探索)。
|
||||
|
||||
**Blocked by:** 01(会话实体与状态机)
|
||||
|
||||
**Status:** ready-for-agent
|
||||
|
||||
- [ ] 活动终态结算时 running 会话转 expired,后续消息 409
|
||||
- [ ] 加速线自动触发被拒并附原因;手动触发可用
|
||||
- [ ] 正式线两种触发来源均可用
|
||||
- [ ] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)
|
||||
12
.scratch/v0.9/issues/08-openclaw-patrol-skill.md
Normal file
12
.scratch/v0.9/issues/08-openclaw-patrol-skill.md
Normal file
@ -0,0 +1,12 @@
|
||||
# 08 — AI 助手巡检/探索 skill + 常驻作业
|
||||
|
||||
**What to build:** 让已集成的 AI 助手成为常驻巡检代理:编写巡检/探索 skill(说明书),指导代理按"调巡检 API → 研判新结果与预算余量 → 决定是否派发探索会话 → 驱动会话对话 → 提交体验记录"的闭环行动,含收到 409 时的收敛行为与衍生变体的种子回溯要求。skill 走部署脚本既有的 skill 同步管线分发;在 AI 助手上注册一个全局常驻 heartbeat/cron 巡检作业(默认节拍 1h)。在 t480 端到端演示:一个配置了种子集的正式线活动,在窗口内被自动巡检并产出探索会话。
|
||||
|
||||
**Blocked by:** 03(巡检 API 契约就绪才能写说明书)
|
||||
|
||||
**Status:** ready-for-agent
|
||||
|
||||
- [ ] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
|
||||
- [ ] 部署同步管线分发 skill(复用现有同步步骤)
|
||||
- [ ] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
|
||||
- [ ] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库
|
||||
113
.scratch/v0.9/spec.md
Normal file
113
.scratch/v0.9/spec.md
Normal file
@ -0,0 +1,113 @@
|
||||
# v0.9 规格说明:探索式评测第一期 —— 周期活动内嵌虚拟用户探索
|
||||
|
||||
**状态**: ready-for-agent
|
||||
**日期**: 2026-08-03
|
||||
**决策依据**: ADR-0003 v2 修订(2026-08-03)、CONTEXT.md「探索式评测」章节、grilling 共识清单(15 项)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
平台的评测能力目前全部建立在"固定场景(考纲)"之上:预设题目、预设规则、统计通过率。这能回答"考纲过了多少",回答不了"真实用户会踩到什么"——真实用户不按考纲出牌,他们会中途改主意、追问、被绕晕、放弃。被评智能体在面对这类复杂拟真行为时的表现(问题诊断与优化建议)目前没有任何评测手段。
|
||||
|
||||
## Solution
|
||||
|
||||
引入与固定场景并行的**探索式评测**模式:OpenClaw 作为**虚拟用户**,按活动配置的种子人设 × 种子目标自主与被评对象对话,产出**体验记录**(目标达成、卡点、被误导处),judge 岗位抽样复核,两条证据线汇入活动报告、v0.7 分析与 Markdown 导出。
|
||||
|
||||
第一期形态为**周期活动内嵌探索**:现有活动骨架不变(静态计划照常执行),全局一个常驻巡检代理(跑在已集成的 AI 助手上,heartbeat/cron 唤醒)按节拍巡检进行中的正式线活动,在预算内派发探索会话。OpenClaw 停摆只暂停探索,固定计划照常完成——耐久性归平台,自主性归代理。
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As an 评测平台用户, I want 创建活动时配置该活动的种子集(种子人设 + 种子目标), so that 虚拟用户的行动有据可依,且同种子集的活动跨期可比
|
||||
2. As an 评测平台用户, I want 创建活动时调整探索预算(或使用平台默认值), so that 探索行为的规模与负载在我的掌控之内
|
||||
3. As an 平台运维者, I want 部署时 AI 助手注册一个全局常驻巡检作业, so that 所有进行中的正式线活动被自动巡检,无需逐活动配置
|
||||
4. As an 常驻代理, I want 每次巡检唤醒时调平台巡检 API 拿到进行中活动、上次巡检以来的新结果与预算余量, so that 我能无状态地自主决定"继续观察 / 派探索会话"
|
||||
5. As an 常驻代理, I want 通过 API 创建探索会话(指定活动、人设、目标、种子出处), so that 我能以虚拟用户身份开始行动
|
||||
6. As an 常驻代理, I want 通过会话 API 发消息并收到被评对象的回复, so that 我能按人设自主决定追问、等待、放弃
|
||||
7. As an 常驻代理, I want 在会话结束时提交结构化体验记录, so that 我的第一手判定(达成/卡点/被误导)被平台留痕
|
||||
8. As an 常驻代理, I want 超出预算时收到 API 明确的 409 拒绝与原因, so that 拒绝本身成为反馈信号,我及时收敛
|
||||
9. As an 常驻代理, I want 在预算内从种子衍生变体(同目标换表达、同人设换追问策略)并被标记衍生出处, so that 行为多样性与留痕兼得
|
||||
10. As an 评测平台用户, I want 在活动报告抽屉看到"探索发现"区块(会话数、目标达成率、问题清单), so that 我知道真实用户遭遇了什么
|
||||
11. As an 评测平台用户, I want 点开单个探索会话查看完整对话与体验记录, so that 每个"问题"都能下钻到原文证据
|
||||
12. As an 分析岗位模型, I want 生成活动分析时收到探索摘要(问题清单 + 达成统计,非全量对话), so that "固定场景成绩"与"真实用户遭遇"两条证据线合成一份结论
|
||||
13. As an judge 岗位模型, I want 对抽样的探索对话做独立复核, so that 体验判定之外有客观质量维度(态度、专业性、幻觉)
|
||||
14. As an 报告导出用户, I want Markdown 导出在存在探索数据时追加"探索发现"区块, so that 离线分享材料完整;无探索数据时导出与旧版一致
|
||||
15. As an 加速调试线用户, I want 手动触发探索会话, so that 我能调试探索流程;同时加速线不被自动巡检打扰(时间压缩与拟真冲突)
|
||||
16. As an 平台, I want OpenClaw 停摆时活动的固定计划照常走完, so that 探索是增强层而非单点故障
|
||||
17. As an 平台, I want 活动窗口结束时仍未关闭的探索会话被妥善结算(标记状态、不再接受消息), so that 不留悬挂会话
|
||||
18. As an 评测平台用户, I want 探索会话与固定场景的子 Run 在报告里清楚区分, so that 通过率口径(ADR-0002)不被探索数据污染
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
### 数据模型
|
||||
|
||||
- 新增**探索会话**独立实体(不并入评测运行):归属活动、指向评测对象、人设(JSON)、目标、种子出处(可空 = 衍生变体须回溯到种子)、状态机(running / completed / failed / expired)、触发来源(auto / manual)、体验记录 JSON、judge 复核 JSON、轮次计数、时间戳
|
||||
- 新增会话轮次表(role、content、latency_ms、created_at),与评测运行的 turns 表平行,不复用
|
||||
- 活动实体新增两个 JSON 配置字段(与 `plan` 同构地位):**种子集**(种子人设数组 + 种子目标数组)、**探索预算**(最大会话数 / 单会话最大轮数 / 会话最小间隔,空则取平台默认)
|
||||
- 活动实体新增 `last_patrolled_at`(巡检 API 每次读取后更新,支撑"上次巡检以来的新结果"语义)
|
||||
- 三个变更各走一个 Alembic 迁移(batch mode)
|
||||
|
||||
### API 契约(全部 X-API-Key 鉴权,沿用 `require_api_key`)
|
||||
|
||||
- `GET /api/exploration/patrol`:返回进行中的正式线活动(time_scale == 1)清单,每项含活动 id/名称/对象、自 `last_patrolled_at` 以来的新结果摘要(复用 `build_campaign_report` 聚合口径)、探索预算余量;响应生成后更新各活动 `last_patrolled_at`
|
||||
- `POST /api/exploration/sessions`:body = campaign_id、persona、goal、seed_ref(可空);平台硬校验(活动存在且 running、正式线仅接受 auto 或 manual 触发;加速线仅接受 manual;预算余量)→ 创建 running 会话;超限 409 + 原因
|
||||
- `POST /api/exploration/sessions/{id}/messages`:body = content;平台转发到目标的通道(复用 ChannelFactory)、持久化双方轮次、返回回复与延迟;单会话轮数超限 409;会话非 running 状态 409
|
||||
- `POST /api/exploration/sessions/{id}/close`:body = 体验记录(goal_achieved: bool、blockers[]、misled[]、emotion、notes);结构校验 + 非法值归一(沿 v0.7 白名单经验);会话转 completed
|
||||
- `GET /api/campaigns/{id}/exploration`:会话列表 + 探索发现聚合(会话数、达成率、问题清单)
|
||||
- 活动窗口 finalize 时:仍 running 的会话转 expired,不再接受消息(在 `resolve_finalize` 同处挂接)
|
||||
|
||||
### 护栏(平台硬执行,不信任客户端自律)
|
||||
|
||||
- 平台默认:≤8 会话/窗口、≤12 轮/会话、相邻会话 ≥30min(正式线真实时间);活动级预算覆盖
|
||||
- 预算计数是平台账本:创建/发消息/关闭均实时校验,超限一律 409
|
||||
|
||||
### 判定双证据线
|
||||
|
||||
- 体验判定:close 接口收结构化自报,为第一手证据
|
||||
- judge 抽样复核:会话结束后平台对对话抽样(默认 ≤3 段,控 token),经 judge 岗位模型产出质量维度复核;ChatClient 可注入(沿 v0.7 分析 seam);异步后台执行,失败落错误不阻塞
|
||||
|
||||
### 报告 / 分析 / 导出
|
||||
|
||||
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);周期对比口径不变(探索数据不参与)
|
||||
- v0.7 分析输入追加探索**摘要**(问题清单 + 达成统计,非全量对话)
|
||||
- Markdown 导出:存在探索数据时追加「探索发现」附录(缺则无痕),附录顺序:智能分析 → 周期对比 → 探索发现
|
||||
|
||||
### 前端
|
||||
|
||||
- 活动创建表单:种子集与预算配置输入(种子可留空 = 该活动不参与探索)
|
||||
- 报告抽屉新增「探索发现」区块(模式沿 v0.7 智能分析区块):达成率、问题清单、会话列表;会话点开查看完整对话 + 体验记录
|
||||
- 不建新顶级页面
|
||||
|
||||
### AI 助手侧(不在平台代码仓内的行为,契约即平台 API)
|
||||
|
||||
- 现有已集成 OpenClaw(AI 助手菜单、openclaw-eval 容器)上扩展巡检/探索 skill,走部署脚本既有 skill 同步管线
|
||||
- 注册一个全局常驻 heartbeat/cron 巡检作业(节拍默认 1h);巡检 API 本身与节拍无关
|
||||
|
||||
### 档位
|
||||
|
||||
- 正式线(time_scale == 1):自动巡检 + 自动派发
|
||||
- 加速调试线:仅手动触发会话,不参与自动巡检
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- 好测试只测外部行为:API 契约、状态机迁移、护栏拒绝、聚合口径、渲染产物;不测内部编排细节
|
||||
- **集成测试(TestClient,主力接缝)**:会话生命周期(创建→对话→关闭)、护栏 409 各分支(超会话数/超轮数/间隔不足/加速线 auto 拒绝)、巡检 API 内容与 last_patrolled_at 推进、finalize 结算 expired、报告/导出纳入。先例:`test_campaigns_api.py`、`test_campaign_comparison_api.py`
|
||||
- **纯函数单测**:探索发现聚合 + Markdown 渲染(dict 进 string 出)。先例:`test_report_render.py`
|
||||
- **假客户端测试**:judge 抽样复核与分析输入扩展,注入假 ChatClient。先例:`test_campaign_analysis.py`
|
||||
- 前端仅 `tsc --noEmit`(无 vitest,沿用现状)
|
||||
- AI 助手 skill 行为不进自动化测试,靠部署后端到端验证(与 v0.4 以来 agenteval-run skill 验收方式一致)
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 双轨之二「探索活动」独立活动类型(OpenClaw 全权接管生命周期)——后续里程碑
|
||||
- 周期对比扩展(探索数据跨期对照)——待种子集指纹与达成率口径稳定
|
||||
- 计划条目重排(原 ADR-0003 v2 字面意义的"自适应重规划")——本期代理只派发探索会话,不改写静态计划
|
||||
- 全局种子库/人设库(活动级配置已够,复用需求出现再提取)
|
||||
- 事件驱动唤醒(纯 heartbeat/cron 巡检)
|
||||
- 前端测试基线(vitest)
|
||||
- volcengine-102 同步
|
||||
|
||||
## Further Notes
|
||||
|
||||
- 词汇表与 ADR 已先行落地(commit c35159f):CONTEXT.md「探索式评测」章节、ADR-0003 v2 修订
|
||||
- 巡检 API 的"新结果摘要"复用 `build_campaign_report` 聚合口径,不重算新指标
|
||||
- OpenClaw heartbeat/cron 事实依据:作业持久化 SQLite、重启不丢、瞬态错误重试、连续 10 次失败自动禁用——天然熔断,与"决策点不可用则跳过"的可靠性边界吻合
|
||||
- 里程碑目录 `.scratch/v0.9/`,票据由 `/to-tickets` 生成于 `issues/`
|
||||
26
CONTEXT.md
26
CONTEXT.md
@ -82,6 +82,32 @@ _Avoid_: 环比(暗示固定自然周期,活动窗口可任意长)
|
||||
活动的两种运行档位,由时间倍速区分:`time_scale == 1` 为正式线(窗口按真实时长走完,代表真实服务周期,享受自动分析、自动周期对比等全链自动化);`time_scale > 1` 为加速调试线(压缩窗口用于快速验证流程,自动化能力默认关闭,仅手动按需触发)。
|
||||
_Avoid_: 真实线、快速模式
|
||||
|
||||
## 探索式评测
|
||||
|
||||
**虚拟用户(Virtual User)**:
|
||||
OpenClaw 在探索式评测中扮演的角色:按人设与探索目标模拟拟真用户行为、与被评对象自主对话的智能体。是"行动者"而非"台词生成器"——对话节奏、追问、放弃均由其自主决定。
|
||||
_Avoid_: 测试员、脚本机器人、模拟器
|
||||
|
||||
**探索式评测(Exploratory Evaluation)**:
|
||||
与固定场景并行的第二种评测模式:不预设考纲,由虚拟用户按种子自由行动,产出"发现的问题"而非"通过率"。固定场景保留为回归基线(可比性、周期对比建在其上),两模式共用活动/窗口/报告外壳,可在同一活动内混编。(决策见 ADR-0003 v2 修订)
|
||||
_Avoid_: 自由测试、压力测试、探索测试
|
||||
|
||||
**探索会话(Exploration Session)**:
|
||||
虚拟用户以"人设 × 目标"组合执行的一段完整对话过程,独立实体存储(`exploration_sessions`,不并入评测运行——Run 绑定场景与规则判定,探索会话两者皆无)。平台提供会话对象、护栏与留痕,对话由 OpenClaw 驱动。
|
||||
_Avoid_: 运行、用例、测试会话
|
||||
|
||||
**种子集(Seed Set)**:
|
||||
活动级配置的种子人设(背景、性格、耐心度)与种子目标(如"完成退货")集合,是探索式评测的可比性单位(对应固定场景的"考纲",与 `plan` 同构)。虚拟用户可在预算约束内从种子衍生有限变体(换表达、换追问策略),全部留痕。
|
||||
_Avoid_: 题库、人设库、剧本
|
||||
|
||||
**体验记录(Experience Record)**:
|
||||
探索会话结束时虚拟用户产出的结构化自报:目标是否达成、卡点、被误导处、情绪变化。探索式评测的第一手证据——意图-结果闭环只有行动者自己能给;judge 岗位对抽样对话独立复核补充质量维度,两条证据链在分析层汇合。
|
||||
_Avoid_: 评测结果、断言、日志
|
||||
|
||||
**巡检(Patrol)**:
|
||||
常驻代理的周期性自主行动:经 OpenClaw heartbeat/cron 唤醒,调平台巡检 API 查看进行中活动的新结果与预算余量,自主决定"继续观察 / 派探索会话"。全局一个常驻巡检作业,无状态地巡检所有活动;OpenClaw 停摆只暂停探索,固定计划照常。
|
||||
_Avoid_: 轮询、心跳(heartbeat 是 OpenClaw 机制名,巡检是平台侧行为)
|
||||
|
||||
## 模型配置
|
||||
|
||||
**模型能力(Capability)**:
|
||||
|
||||
1
backend/agenteval/exploration/__init__.py
Normal file
1
backend/agenteval/exploration/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Exploratory evaluation (探索式评测): virtual-user sessions inside campaigns."""
|
||||
107
backend/agenteval/exploration/models.py
Normal file
107
backend/agenteval/exploration/models.py
Normal file
@ -0,0 +1,107 @@
|
||||
"""Domain models and budget defaults for exploratory evaluation (v0.9).
|
||||
|
||||
Exploration sessions are an independent entity — never merged into EvalRun —
|
||||
so pass-rate semantics (ADR-0002) and scenario comparability (ADR-0001) stay
|
||||
untouched. Budget enforcement is a platform ledger: the defaults here apply
|
||||
unless overridden per-campaign via ``Campaign.exploration_budget``; the
|
||||
enforcement itself always happens server-side.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Any, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from agenteval.models import Campaign
|
||||
|
||||
DEFAULT_MAX_SESSIONS_PER_WINDOW = 8
|
||||
DEFAULT_MAX_TURNS_PER_SESSION = 12
|
||||
DEFAULT_MIN_SESSION_INTERVAL_SECONDS = 30 * 60
|
||||
|
||||
VALID_EMOTIONS = ("positive", "neutral", "confused", "frustrated")
|
||||
|
||||
|
||||
class ExplorationSessionStatus(str, Enum):
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
EXPIRED = "expired"
|
||||
|
||||
|
||||
class ExplorationTrigger(str, Enum):
|
||||
AUTO = "auto"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class ExplorationMessage(BaseModel):
|
||||
"""One chat message (role/content) inside an exploration session."""
|
||||
|
||||
id: Optional[str] = None
|
||||
session_id: str
|
||||
round_index: int = 0
|
||||
role: str = "user"
|
||||
content: str = ""
|
||||
latency_ms: Optional[int] = None
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ExplorationSession(BaseModel):
|
||||
"""A virtual-user exploration session belonging to one campaign."""
|
||||
|
||||
id: Optional[str] = None
|
||||
campaign_id: str
|
||||
target_id: str
|
||||
persona: dict[str, Any] = Field(default_factory=dict)
|
||||
goal: str = ""
|
||||
seed_ref: Optional[dict[str, Any]] = None
|
||||
status: ExplorationSessionStatus = ExplorationSessionStatus.RUNNING
|
||||
triggered_by: ExplorationTrigger = ExplorationTrigger.AUTO
|
||||
experience: Optional[dict[str, Any]] = None
|
||||
judge_review: Optional[dict[str, Any]] = None
|
||||
turn_count: int = 0
|
||||
error: Optional[str] = None
|
||||
created_at: Optional[datetime] = None
|
||||
closed_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ExplorationBudget(BaseModel):
|
||||
max_sessions: int = DEFAULT_MAX_SESSIONS_PER_WINDOW
|
||||
max_turns: int = DEFAULT_MAX_TURNS_PER_SESSION
|
||||
min_interval_seconds: int = DEFAULT_MIN_SESSION_INTERVAL_SECONDS
|
||||
|
||||
|
||||
def resolve_budget(campaign: Campaign) -> ExplorationBudget:
|
||||
"""Effective exploration budget for a campaign.
|
||||
|
||||
Platform defaults with per-field campaign overrides; unset override
|
||||
fields fall back to the defaults.
|
||||
"""
|
||||
override = campaign.exploration_budget
|
||||
if override is None:
|
||||
return ExplorationBudget()
|
||||
return ExplorationBudget(
|
||||
max_sessions=override.max_sessions or DEFAULT_MAX_SESSIONS_PER_WINDOW,
|
||||
max_turns=override.max_turns or DEFAULT_MAX_TURNS_PER_SESSION,
|
||||
min_interval_seconds=override.min_interval_seconds or DEFAULT_MIN_SESSION_INTERVAL_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
def normalize_experience(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Whitelist-normalize a self-reported experience record (v0.7 precedent).
|
||||
|
||||
Invalid values are coerced to safe defaults rather than rejected, so a
|
||||
sloppy virtual user still leaves a usable evidence row.
|
||||
"""
|
||||
raw_blockers = raw.get("blockers") if isinstance(raw.get("blockers"), list) else []
|
||||
raw_misled = raw.get("misled") if isinstance(raw.get("misled"), list) else []
|
||||
blockers = [str(item) for item in raw_blockers if isinstance(item, (str, int, float))]
|
||||
misled = [str(item) for item in raw_misled if isinstance(item, (str, int, float))]
|
||||
emotion = raw.get("emotion")
|
||||
return {
|
||||
"goal_achieved": bool(raw.get("goal_achieved")),
|
||||
"blockers": blockers,
|
||||
"misled": misled,
|
||||
"emotion": emotion if emotion in VALID_EMOTIONS else "neutral",
|
||||
"notes": str(raw.get("notes") or ""),
|
||||
}
|
||||
@ -271,6 +271,27 @@ class CampaignSummary(BaseModel):
|
||||
scheduler: SchedulerState = Field(default_factory=SchedulerState)
|
||||
|
||||
|
||||
class ExplorationSeeds(BaseModel):
|
||||
"""Seed set (种子集) for exploratory evaluation: seed personas × seed goals.
|
||||
|
||||
The comparability unit for exploratory evaluation — campaigns sharing a
|
||||
seed set are comparable across periods. Empty/absent means the campaign
|
||||
opts out of exploration.
|
||||
"""
|
||||
|
||||
personas: list[str] = Field(default_factory=list)
|
||||
goals: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ExplorationBudgetConfig(BaseModel):
|
||||
"""Per-campaign exploration budget override; unset fields fall back to
|
||||
platform defaults at enforcement time."""
|
||||
|
||||
max_sessions: Optional[int] = Field(default=None, gt=0)
|
||||
max_turns: Optional[int] = Field(default=None, gt=0)
|
||||
min_interval_seconds: Optional[int] = Field(default=None, gt=0)
|
||||
|
||||
|
||||
class Campaign(BaseModel):
|
||||
"""An evaluation campaign: a service-cycle window over a single target,
|
||||
driving many child Runs from a static plan (ADR-0003)."""
|
||||
@ -290,6 +311,9 @@ class Campaign(BaseModel):
|
||||
created_at: Optional[datetime] = None
|
||||
summary: Optional[CampaignSummary] = None
|
||||
analysis_model_config_id: Optional[str] = None
|
||||
exploration_seeds: Optional[ExplorationSeeds] = None
|
||||
exploration_budget: Optional[ExplorationBudgetConfig] = None
|
||||
last_patrolled_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class Turn(BaseModel):
|
||||
|
||||
@ -27,6 +27,15 @@ def utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def as_utc(dt: datetime) -> datetime:
|
||||
"""Attach UTC tzinfo to a naive datetime.
|
||||
|
||||
SQLite round-trips drop tzinfo; stored times are always UTC, so a naive
|
||||
value read back is restored as UTC before any comparison with utc_now().
|
||||
"""
|
||||
return dt if dt.tzinfo is not None else dt.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def iso_utc(dt: datetime | None) -> str | None:
|
||||
"""Serialize a datetime to ISO 8601 with UTC timezone suffix.
|
||||
|
||||
@ -188,6 +197,9 @@ class CampaignDB(SQLModel, table=True):
|
||||
created_at: Optional[datetime] = Field(default_factory=utc_now)
|
||||
summary: Optional[str] = None
|
||||
analysis_model_config_id: Optional[str] = None
|
||||
exploration_seeds: Optional[str] = None
|
||||
exploration_budget: Optional[str] = None
|
||||
last_patrolled_at: Optional[datetime] = None
|
||||
|
||||
def get_plan(self) -> list[dict[str, Any]]:
|
||||
return _json_loads(self.plan)
|
||||
@ -201,6 +213,18 @@ class CampaignDB(SQLModel, table=True):
|
||||
def set_summary(self, summary: dict[str, Any]) -> None:
|
||||
self.summary = _json_dumps(summary)
|
||||
|
||||
def get_exploration_seeds(self) -> Optional[dict[str, Any]]:
|
||||
return _json_loads(self.exploration_seeds) if self.exploration_seeds else None
|
||||
|
||||
def set_exploration_seeds(self, seeds: dict[str, Any]) -> None:
|
||||
self.exploration_seeds = _json_dumps(seeds)
|
||||
|
||||
def get_exploration_budget(self) -> Optional[dict[str, Any]]:
|
||||
return _json_loads(self.exploration_budget) if self.exploration_budget else None
|
||||
|
||||
def set_exploration_budget(self, budget: dict[str, Any]) -> None:
|
||||
self.exploration_budget = _json_dumps(budget)
|
||||
|
||||
|
||||
class CampaignAnalysisDB(SQLModel, table=True):
|
||||
"""One row per campaign holding its intelligent analysis (智能分析) state."""
|
||||
@ -251,6 +275,75 @@ class CampaignPeriodComparisonDB(SQLModel, table=True):
|
||||
self.result = _json_dumps(result)
|
||||
|
||||
|
||||
class ExplorationSessionDB(SQLModel, table=True):
|
||||
"""One virtual-user exploration session (探索会话) within a campaign.
|
||||
|
||||
Independent entity, deliberately NOT an EvalRun: exploration outcomes feed
|
||||
the 体验判定 evidence line and must not pollute pass-rate semantics
|
||||
(ADR-0001 comparability / ADR-0002).
|
||||
"""
|
||||
|
||||
__tablename__ = "exploration_sessions"
|
||||
|
||||
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
|
||||
campaign_id: str = Field(index=True, foreign_key="campaigns.id")
|
||||
target_id: str = Field(foreign_key="eval_targets.id")
|
||||
persona: str = "{}"
|
||||
goal: str = ""
|
||||
seed_ref: Optional[str] = None
|
||||
status: str = "running"
|
||||
triggered_by: str = "auto"
|
||||
experience: Optional[str] = None
|
||||
judge_review: Optional[str] = None
|
||||
turn_count: int = 0
|
||||
error: Optional[str] = None
|
||||
created_at: Optional[datetime] = Field(default_factory=utc_now)
|
||||
closed_at: Optional[datetime] = None
|
||||
|
||||
def get_persona(self) -> dict[str, Any]:
|
||||
return _json_loads(self.persona)
|
||||
|
||||
def set_persona(self, persona: dict[str, Any]) -> None:
|
||||
self.persona = _json_dumps(persona)
|
||||
|
||||
def get_seed_ref(self) -> Optional[dict[str, Any]]:
|
||||
return _json_loads(self.seed_ref) if self.seed_ref else None
|
||||
|
||||
def set_seed_ref(self, seed_ref: dict[str, Any]) -> None:
|
||||
self.seed_ref = _json_dumps(seed_ref)
|
||||
|
||||
def get_experience(self) -> Optional[dict[str, Any]]:
|
||||
return _json_loads(self.experience) if self.experience else None
|
||||
|
||||
def set_experience(self, experience: dict[str, Any]) -> None:
|
||||
self.experience = _json_dumps(experience)
|
||||
|
||||
def get_judge_review(self) -> Optional[dict[str, Any]]:
|
||||
return _json_loads(self.judge_review) if self.judge_review else None
|
||||
|
||||
def set_judge_review(self, judge_review: dict[str, Any]) -> None:
|
||||
self.judge_review = _json_dumps(judge_review)
|
||||
|
||||
|
||||
class ExplorationMessageDB(SQLModel, table=True):
|
||||
"""One chat message inside an exploration session (role/content form).
|
||||
|
||||
Parallel to ``eval_runs`` turns but never shared with them. A completed
|
||||
question-answer pair contributes two rows (user + assistant); the reply
|
||||
row carries ``latency_ms``.
|
||||
"""
|
||||
|
||||
__tablename__ = "exploration_messages"
|
||||
|
||||
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
|
||||
session_id: str = Field(index=True, foreign_key="exploration_sessions.id")
|
||||
round_index: int = 0
|
||||
role: str = "user"
|
||||
content: str = ""
|
||||
latency_ms: Optional[int] = None
|
||||
created_at: Optional[datetime] = Field(default_factory=utc_now)
|
||||
|
||||
|
||||
class EvalRunDB(SQLModel, table=True):
|
||||
"""Database table for evaluation runs."""
|
||||
|
||||
|
||||
@ -4,6 +4,7 @@ from typing import Generic, Optional, TypeVar
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from agenteval.exploration.models import ExplorationMessage, ExplorationSession
|
||||
from agenteval.models import Campaign, Case, EvalResult, EvalRun, EvalTarget, Scenario
|
||||
from agenteval.services.model_configs import ModelConfigService
|
||||
from agenteval.storage.db import (
|
||||
@ -13,6 +14,8 @@ from agenteval.storage.db import (
|
||||
EvalResultDB,
|
||||
EvalRunDB,
|
||||
EvalTargetDB,
|
||||
ExplorationMessageDB,
|
||||
ExplorationSessionDB,
|
||||
ScenarioDB,
|
||||
TurnDB,
|
||||
get_session,
|
||||
@ -282,11 +285,7 @@ class RunRepository(BaseRepository[EvalRun, EvalRunDB]):
|
||||
)
|
||||
|
||||
def list_by_campaign(self, campaign_id: str) -> list[EvalRun]:
|
||||
statement = (
|
||||
select(EvalRunDB)
|
||||
.where(EvalRunDB.campaign_id == campaign_id)
|
||||
.order_by(EvalRunDB.started_at)
|
||||
)
|
||||
statement = select(EvalRunDB).where(EvalRunDB.campaign_id == campaign_id).order_by(EvalRunDB.started_at)
|
||||
return [self._from_db(r) for r in self.session.exec(statement).all()]
|
||||
|
||||
def mark_orphans_failed(self) -> int:
|
||||
@ -353,10 +352,15 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
|
||||
completed_at=campaign.completed_at,
|
||||
created_at=campaign.created_at,
|
||||
analysis_model_config_id=campaign.analysis_model_config_id,
|
||||
last_patrolled_at=campaign.last_patrolled_at,
|
||||
)
|
||||
db.set_plan([entry.model_dump(mode="json") for entry in campaign.plan])
|
||||
if campaign.summary:
|
||||
db.set_summary(campaign.summary.model_dump(mode="json"))
|
||||
if campaign.exploration_seeds is not None:
|
||||
db.set_exploration_seeds(campaign.exploration_seeds.model_dump(mode="json"))
|
||||
if campaign.exploration_budget is not None:
|
||||
db.set_exploration_budget(campaign.exploration_budget.model_dump(mode="json"))
|
||||
return db
|
||||
|
||||
def _from_db(self, db: CampaignDB) -> Campaign:
|
||||
@ -373,6 +377,9 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
|
||||
created_at=db.created_at,
|
||||
summary=db.get_summary(),
|
||||
analysis_model_config_id=db.analysis_model_config_id,
|
||||
exploration_seeds=db.get_exploration_seeds(),
|
||||
exploration_budget=db.get_exploration_budget(),
|
||||
last_patrolled_at=db.last_patrolled_at,
|
||||
)
|
||||
|
||||
def update(self, campaign: Campaign) -> Optional[Campaign]:
|
||||
@ -388,8 +395,13 @@ class CampaignRepository(BaseRepository[Campaign, CampaignDB]):
|
||||
existing.started_at = campaign.started_at
|
||||
existing.completed_at = campaign.completed_at
|
||||
existing.analysis_model_config_id = campaign.analysis_model_config_id
|
||||
existing.last_patrolled_at = campaign.last_patrolled_at
|
||||
if campaign.summary is not None:
|
||||
existing.set_summary(campaign.summary.model_dump(mode="json"))
|
||||
if campaign.exploration_seeds is not None:
|
||||
existing.set_exploration_seeds(campaign.exploration_seeds.model_dump(mode="json"))
|
||||
if campaign.exploration_budget is not None:
|
||||
existing.set_exploration_budget(campaign.exploration_budget.model_dump(mode="json"))
|
||||
self.session.add(existing)
|
||||
self.session.commit()
|
||||
self.session.refresh(existing)
|
||||
@ -441,9 +453,7 @@ class CampaignPeriodComparisonRepository:
|
||||
self.session = session or get_session()
|
||||
|
||||
def get_by_campaign(self, campaign_id: str) -> Optional[CampaignPeriodComparisonDB]:
|
||||
statement = select(CampaignPeriodComparisonDB).where(
|
||||
CampaignPeriodComparisonDB.campaign_id == campaign_id
|
||||
)
|
||||
statement = select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == campaign_id)
|
||||
return self.session.exec(statement).first()
|
||||
|
||||
def upsert(
|
||||
@ -510,3 +520,113 @@ class ResultRepository:
|
||||
self.session.commit()
|
||||
self.session.refresh(db)
|
||||
return _result_from_db(db)
|
||||
|
||||
|
||||
class ExplorationSessionRepository(BaseRepository[ExplorationSession, ExplorationSessionDB]):
|
||||
"""Repository for virtual-user exploration sessions (探索会话)."""
|
||||
|
||||
_table = ExplorationSessionDB
|
||||
_order_by = "created_at"
|
||||
|
||||
def _copy_mutable(self, db: ExplorationSessionDB, session_obj: ExplorationSession) -> None:
|
||||
db.goal = session_obj.goal
|
||||
db.status = session_obj.status.value
|
||||
db.triggered_by = session_obj.triggered_by.value
|
||||
db.turn_count = session_obj.turn_count
|
||||
db.error = session_obj.error
|
||||
db.closed_at = session_obj.closed_at
|
||||
db.set_persona(session_obj.persona)
|
||||
if session_obj.seed_ref is not None:
|
||||
db.set_seed_ref(session_obj.seed_ref)
|
||||
if session_obj.experience is not None:
|
||||
db.set_experience(session_obj.experience)
|
||||
if session_obj.judge_review is not None:
|
||||
db.set_judge_review(session_obj.judge_review)
|
||||
|
||||
def _to_db(self, session_obj: ExplorationSession) -> ExplorationSessionDB:
|
||||
db = ExplorationSessionDB(
|
||||
id=session_obj.id,
|
||||
campaign_id=session_obj.campaign_id,
|
||||
target_id=session_obj.target_id,
|
||||
created_at=session_obj.created_at,
|
||||
)
|
||||
self._copy_mutable(db, session_obj)
|
||||
return db
|
||||
|
||||
def _from_db(self, db: ExplorationSessionDB) -> ExplorationSession:
|
||||
return ExplorationSession(
|
||||
id=db.id,
|
||||
campaign_id=db.campaign_id,
|
||||
target_id=db.target_id,
|
||||
persona=db.get_persona(),
|
||||
goal=db.goal,
|
||||
seed_ref=db.get_seed_ref(),
|
||||
status=db.status,
|
||||
triggered_by=db.triggered_by,
|
||||
experience=db.get_experience(),
|
||||
judge_review=db.get_judge_review(),
|
||||
turn_count=db.turn_count,
|
||||
error=db.error,
|
||||
created_at=db.created_at,
|
||||
closed_at=db.closed_at,
|
||||
)
|
||||
|
||||
def update(self, session_obj: ExplorationSession) -> Optional[ExplorationSession]:
|
||||
existing = self.session.get(ExplorationSessionDB, session_obj.id)
|
||||
if not existing:
|
||||
return None
|
||||
self._copy_mutable(existing, session_obj)
|
||||
self.session.add(existing)
|
||||
self.session.commit()
|
||||
self.session.refresh(existing)
|
||||
return self._from_db(existing)
|
||||
|
||||
def list_by_campaign(self, campaign_id: str) -> list[ExplorationSession]:
|
||||
statement = (
|
||||
select(ExplorationSessionDB)
|
||||
.where(ExplorationSessionDB.campaign_id == campaign_id)
|
||||
.order_by(ExplorationSessionDB.created_at)
|
||||
)
|
||||
return [self._from_db(r) for r in self.session.exec(statement).all()]
|
||||
|
||||
|
||||
class ExplorationMessageRepository:
|
||||
"""Append-only repository for exploration session chat rows."""
|
||||
|
||||
def __init__(self, session: Optional[Session] = None):
|
||||
self.session = session or get_session()
|
||||
|
||||
def save_message(self, message: ExplorationMessage) -> ExplorationMessage:
|
||||
db = ExplorationMessageDB(
|
||||
id=message.id,
|
||||
session_id=message.session_id,
|
||||
round_index=message.round_index,
|
||||
role=message.role,
|
||||
content=message.content,
|
||||
latency_ms=message.latency_ms,
|
||||
created_at=message.created_at,
|
||||
)
|
||||
self.session.add(db)
|
||||
self.session.commit()
|
||||
self.session.refresh(db)
|
||||
message.id = db.id
|
||||
return message
|
||||
|
||||
def list_by_session(self, session_id: str) -> list[ExplorationMessage]:
|
||||
statement = (
|
||||
select(ExplorationMessageDB)
|
||||
.where(ExplorationMessageDB.session_id == session_id)
|
||||
.order_by(ExplorationMessageDB.created_at)
|
||||
)
|
||||
return [
|
||||
ExplorationMessage(
|
||||
id=r.id,
|
||||
session_id=r.session_id,
|
||||
round_index=r.round_index,
|
||||
role=r.role,
|
||||
content=r.content,
|
||||
latency_ms=r.latency_ms,
|
||||
created_at=r.created_at,
|
||||
)
|
||||
for r in self.session.exec(statement).all()
|
||||
]
|
||||
|
||||
@ -13,7 +13,19 @@ from agenteval.storage.db import get_session, init_db
|
||||
from agenteval.storage.repository import RunRepository
|
||||
from agenteval.version import get_build_info, get_version
|
||||
from agenteval.web.deps import require_api_key
|
||||
from agenteval.web.routers import auth, campaigns, files, model_configs, proxy, reports, runs, scenarios, stats, targets
|
||||
from agenteval.web.routers import (
|
||||
auth,
|
||||
campaigns,
|
||||
exploration,
|
||||
files,
|
||||
model_configs,
|
||||
proxy,
|
||||
reports,
|
||||
runs,
|
||||
scenarios,
|
||||
stats,
|
||||
targets,
|
||||
)
|
||||
from agenteval.web.websocket import ws_manager
|
||||
|
||||
|
||||
@ -74,6 +86,7 @@ app.include_router(targets.router, prefix="/api/targets", tags=["targets"], depe
|
||||
app.include_router(scenarios.router, prefix="/api/scenarios", tags=["scenarios"], dependencies=_api_deps)
|
||||
app.include_router(runs.router, prefix="/api/runs", tags=["runs"], dependencies=_api_deps)
|
||||
app.include_router(campaigns.router, prefix="/api/campaigns", tags=["campaigns"], dependencies=_api_deps)
|
||||
app.include_router(exploration.router, prefix="/api/exploration", tags=["exploration"], dependencies=_api_deps)
|
||||
app.include_router(reports.router, prefix="/api/reports", tags=["reports"], dependencies=_api_deps)
|
||||
app.include_router(stats.router, prefix="/api/stats", tags=["stats"], dependencies=_api_deps)
|
||||
app.include_router(files.router, prefix="/api/files", tags=["files"], dependencies=_api_deps)
|
||||
|
||||
@ -26,7 +26,7 @@ from agenteval.evaluation.report import (
|
||||
summarize_campaign_progress,
|
||||
)
|
||||
from agenteval.evaluation.report_render import render_campaign_markdown
|
||||
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus
|
||||
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, ExplorationBudgetConfig, ExplorationSeeds
|
||||
from agenteval.storage.db import iso_utc, utc_now
|
||||
from agenteval.storage.model_config_repository import ModelConfigRepository
|
||||
from agenteval.storage.repository import (
|
||||
@ -49,6 +49,8 @@ class CreateCampaignRequest(BaseModel):
|
||||
time_scale: float = Field(default=1.0, gt=0)
|
||||
plan: list[CampaignPlanEntry] = Field(min_length=1)
|
||||
analysis_model_config_id: str | None = None
|
||||
exploration_seeds: ExplorationSeeds | None = None
|
||||
exploration_budget: ExplorationBudgetConfig | None = None
|
||||
|
||||
|
||||
@router.get("")
|
||||
@ -84,6 +86,10 @@ async def create_campaign(
|
||||
if not ModelConfigRepository(session).get(request.analysis_model_config_id):
|
||||
raise HTTPException(status_code=400, detail="analysis model config not found")
|
||||
|
||||
seeds = request.exploration_seeds
|
||||
if seeds is not None and not seeds.personas and not seeds.goals:
|
||||
seeds = None # 种子留空 = 该活动不参与探索
|
||||
|
||||
campaign = Campaign(
|
||||
name=request.name,
|
||||
target_id=request.target_id,
|
||||
@ -91,6 +97,8 @@ async def create_campaign(
|
||||
time_scale=request.time_scale,
|
||||
plan=request.plan,
|
||||
analysis_model_config_id=request.analysis_model_config_id,
|
||||
exploration_seeds=seeds,
|
||||
exploration_budget=request.exploration_budget,
|
||||
)
|
||||
repo = CampaignRepository(session)
|
||||
campaign = repo.create(campaign)
|
||||
|
||||
293
backend/agenteval/web/routers/exploration.py
Normal file
293
backend/agenteval/web/routers/exploration.py
Normal file
@ -0,0 +1,293 @@
|
||||
"""API routes for exploratory evaluation sessions (探索式评测, v0.9).
|
||||
|
||||
The virtual user (OpenClaw) drives these sessions through plain HTTP: create a
|
||||
running session against a campaign, converse with the target through its real
|
||||
channel, then close with a structured self-reported experience record.
|
||||
|
||||
Guardrails are a platform ledger — enforced here on every call, never trusted
|
||||
to client self-discipline. Budget overruns and state violations are rejected
|
||||
with 409 plus a readable reason, so the rejection itself is feedback to the
|
||||
resident agent.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlmodel import Session
|
||||
|
||||
from agenteval.channels.factory import ChannelFactory
|
||||
from agenteval.config import get_settings
|
||||
from agenteval.evaluation.report import generate_campaign_report
|
||||
from agenteval.exploration.models import (
|
||||
ExplorationBudget,
|
||||
ExplorationMessage,
|
||||
ExplorationSession,
|
||||
ExplorationSessionStatus,
|
||||
ExplorationTrigger,
|
||||
normalize_experience,
|
||||
resolve_budget,
|
||||
)
|
||||
from agenteval.models import Campaign, CampaignStatus, EvalRun
|
||||
from agenteval.storage.db import as_utc, iso_utc, utc_now
|
||||
from agenteval.storage.repository import (
|
||||
CampaignRepository,
|
||||
ExplorationMessageRepository,
|
||||
ExplorationSessionRepository,
|
||||
RunRepository,
|
||||
ScenarioRepository,
|
||||
TargetRepository,
|
||||
)
|
||||
from agenteval.web.deps import get_db
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class CreateSessionRequest(BaseModel):
|
||||
campaign_id: str
|
||||
persona: dict[str, Any]
|
||||
goal: str = Field(min_length=1)
|
||||
seed_ref: dict[str, Any] | None = None
|
||||
triggered_by: ExplorationTrigger = ExplorationTrigger.AUTO
|
||||
|
||||
|
||||
class SendMessageRequest(BaseModel):
|
||||
content: str = Field(min_length=1)
|
||||
|
||||
|
||||
class CloseSessionRequest(BaseModel):
|
||||
experience: dict[str, Any]
|
||||
|
||||
|
||||
def _check_creation_guardrails(
|
||||
campaign: Campaign,
|
||||
triggered_by: ExplorationTrigger,
|
||||
budget: ExplorationBudget,
|
||||
repo: ExplorationSessionRepository,
|
||||
) -> None:
|
||||
if campaign.status != CampaignStatus.RUNNING:
|
||||
raise HTTPException(status_code=409, detail="活动不在进行中,无法创建探索会话")
|
||||
if campaign.time_scale != 1 and triggered_by != ExplorationTrigger.MANUAL:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="加速调试线仅允许手动创建探索会话(时间压缩与拟真相冲突)",
|
||||
)
|
||||
sessions = repo.list_by_campaign(campaign.id)
|
||||
if len(sessions) >= budget.max_sessions:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"探索会话数超出预算:本活动窗口最多 {budget.max_sessions} 个会话",
|
||||
)
|
||||
if sessions:
|
||||
latest = max(s.created_at for s in sessions if s.created_at)
|
||||
elapsed = (utc_now() - as_utc(latest)).total_seconds()
|
||||
if elapsed < budget.min_interval_seconds:
|
||||
wait_minutes = budget.min_interval_seconds // 60
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"相邻探索会话间隔不足:最小间隔 {wait_minutes} 分钟,请稍后再试",
|
||||
)
|
||||
|
||||
|
||||
def _new_runs_since(runs: list[EvalRun], watermark: datetime | None) -> list[EvalRun]:
|
||||
fresh = []
|
||||
for run in runs:
|
||||
if run.completed_at is None:
|
||||
continue
|
||||
if watermark is not None and as_utc(run.completed_at) <= watermark:
|
||||
continue
|
||||
fresh.append(run)
|
||||
return fresh
|
||||
|
||||
|
||||
@router.get("/patrol")
|
||||
async def patrol(session: Session = Depends(get_db)) -> dict:
|
||||
"""Stateless patrol for the resident agent.
|
||||
|
||||
Reports every running production-line (time_scale == 1) campaign that
|
||||
participates in exploration (has a seed set), with new results since the
|
||||
last watermark and the remaining exploration budget. Advances each
|
||||
patrolled campaign's watermark after building the response, so the next
|
||||
call only reports increments.
|
||||
"""
|
||||
campaign_repo = CampaignRepository(session)
|
||||
run_repo = RunRepository(session)
|
||||
exploration_repo = ExplorationSessionRepository(session)
|
||||
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
|
||||
target_names = {t.id: t.name for t in TargetRepository(session).list_all()}
|
||||
|
||||
patrolled_at = utc_now()
|
||||
entries: list[dict[str, Any]] = []
|
||||
patrolled_campaigns: list[Campaign] = []
|
||||
for campaign in campaign_repo.list_all():
|
||||
if campaign.status != CampaignStatus.RUNNING:
|
||||
continue
|
||||
if campaign.time_scale != 1:
|
||||
continue
|
||||
if campaign.exploration_seeds is None:
|
||||
continue
|
||||
|
||||
watermark = as_utc(campaign.last_patrolled_at) if campaign.last_patrolled_at else None
|
||||
fresh = _new_runs_since(run_repo.list_by_campaign(campaign.id), watermark)
|
||||
new_results = None
|
||||
if fresh:
|
||||
report = generate_campaign_report(campaign, fresh, scenario_names=scenario_names)
|
||||
new_results = {
|
||||
"summary": report["summary"],
|
||||
"capability_summary": report["capability_summary"],
|
||||
}
|
||||
|
||||
budget = resolve_budget(campaign)
|
||||
sessions = exploration_repo.list_by_campaign(campaign.id)
|
||||
seconds_since_last_session = None
|
||||
if sessions:
|
||||
latest = max(as_utc(s.created_at) for s in sessions if s.created_at)
|
||||
seconds_since_last_session = int((patrolled_at - latest).total_seconds())
|
||||
|
||||
entries.append(
|
||||
{
|
||||
"campaign_id": campaign.id,
|
||||
"campaign_name": campaign.name,
|
||||
"target_id": campaign.target_id,
|
||||
"target_name": target_names.get(campaign.target_id),
|
||||
"last_patrolled_at": iso_utc(campaign.last_patrolled_at),
|
||||
"new_results": new_results,
|
||||
"budget": {
|
||||
"max_sessions": budget.max_sessions,
|
||||
"sessions_used": len(sessions),
|
||||
"remaining_sessions": max(0, budget.max_sessions - len(sessions)),
|
||||
"max_turns": budget.max_turns,
|
||||
"min_interval_seconds": budget.min_interval_seconds,
|
||||
"seconds_since_last_session": seconds_since_last_session,
|
||||
},
|
||||
}
|
||||
)
|
||||
patrolled_campaigns.append(campaign)
|
||||
|
||||
# 水位取构建响应之后的时刻:查询与持久化之间完成的结果不会在下次重复上报。
|
||||
watermark_at = utc_now()
|
||||
for campaign in patrolled_campaigns:
|
||||
campaign.last_patrolled_at = watermark_at
|
||||
campaign_repo.update(campaign)
|
||||
|
||||
return {"patrolled_at": iso_utc(watermark_at), "campaigns": entries}
|
||||
|
||||
|
||||
@router.post("/sessions")
|
||||
async def create_session(
|
||||
request: CreateSessionRequest,
|
||||
session: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
campaign = CampaignRepository(session).get(request.campaign_id)
|
||||
if not campaign:
|
||||
raise HTTPException(status_code=404, detail="campaign not found")
|
||||
if not TargetRepository(session).get(campaign.target_id):
|
||||
raise HTTPException(status_code=404, detail="campaign target not found")
|
||||
|
||||
budget = resolve_budget(campaign)
|
||||
repo = ExplorationSessionRepository(session)
|
||||
_check_creation_guardrails(campaign, request.triggered_by, budget, repo)
|
||||
|
||||
session_obj = ExplorationSession(
|
||||
campaign_id=campaign.id,
|
||||
target_id=campaign.target_id,
|
||||
persona=request.persona,
|
||||
goal=request.goal,
|
||||
seed_ref=request.seed_ref,
|
||||
triggered_by=request.triggered_by,
|
||||
)
|
||||
return repo.create(session_obj).model_dump(mode="json")
|
||||
|
||||
|
||||
@router.post("/sessions/{session_id}/messages")
|
||||
async def send_session_message(
|
||||
session_id: str,
|
||||
request: SendMessageRequest,
|
||||
session: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
repo = ExplorationSessionRepository(session)
|
||||
session_obj = repo.get(session_id)
|
||||
if not session_obj:
|
||||
raise HTTPException(status_code=404, detail="exploration session not found")
|
||||
if session_obj.status != ExplorationSessionStatus.RUNNING:
|
||||
raise HTTPException(status_code=409, detail="探索会话不在进行中,拒收消息")
|
||||
|
||||
campaign = CampaignRepository(session).get(session_obj.campaign_id)
|
||||
budget = resolve_budget(campaign) if campaign else ExplorationBudget()
|
||||
if session_obj.turn_count >= budget.max_turns:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"会话轮数超出预算:单会话最多 {budget.max_turns} 轮",
|
||||
)
|
||||
|
||||
target = TargetRepository(session).get(session_obj.target_id)
|
||||
if not target:
|
||||
raise HTTPException(status_code=404, detail="session target not found")
|
||||
|
||||
channel = ChannelFactory.create(target)
|
||||
sent_at = utc_now()
|
||||
try:
|
||||
send_result = await channel.send(request.content)
|
||||
except Exception as exc: # channel adapters raise transport-specific errors
|
||||
raise HTTPException(status_code=502, detail=f"评测对象通道发送失败: {exc}")
|
||||
if not send_result.ok:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"评测对象通道发送失败: {send_result.error}",
|
||||
)
|
||||
|
||||
# 消息已送达即消耗一轮预算(平台账本):先落用户消息,再等回复。
|
||||
message_repo = ExplorationMessageRepository(session)
|
||||
round_index = session_obj.turn_count + 1
|
||||
message_repo.save_message(
|
||||
ExplorationMessage(
|
||||
session_id=session_obj.id, round_index=round_index, role="user", content=request.content, created_at=sent_at
|
||||
)
|
||||
)
|
||||
session_obj.turn_count = round_index
|
||||
repo.update(session_obj)
|
||||
|
||||
try:
|
||||
reply = await channel.poll_reply(
|
||||
send_result.question_msg_id,
|
||||
timeout=get_settings().poll_reply_timeout,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=502, detail=f"等待评测对象回复失败: {exc}")
|
||||
if reply is None:
|
||||
raise HTTPException(status_code=502, detail="等待评测对象回复超时")
|
||||
|
||||
received_at = utc_now()
|
||||
latency_ms = int((received_at - sent_at).total_seconds() * 1000)
|
||||
reply_text = str(reply.content)
|
||||
message_repo.save_message(
|
||||
ExplorationMessage(
|
||||
session_id=session_obj.id,
|
||||
round_index=round_index,
|
||||
role="assistant",
|
||||
content=reply_text,
|
||||
latency_ms=latency_ms,
|
||||
created_at=received_at,
|
||||
)
|
||||
)
|
||||
return {"reply": reply_text, "latency_ms": latency_ms, "turn_count": round_index}
|
||||
|
||||
|
||||
@router.post("/sessions/{session_id}/close")
|
||||
async def close_session(
|
||||
session_id: str,
|
||||
request: CloseSessionRequest,
|
||||
session: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
repo = ExplorationSessionRepository(session)
|
||||
session_obj = repo.get(session_id)
|
||||
if not session_obj:
|
||||
raise HTTPException(status_code=404, detail="exploration session not found")
|
||||
if session_obj.status != ExplorationSessionStatus.RUNNING:
|
||||
raise HTTPException(status_code=409, detail="探索会话不在进行中,无法关闭")
|
||||
|
||||
session_obj.experience = normalize_experience(request.experience)
|
||||
session_obj.status = ExplorationSessionStatus.COMPLETED
|
||||
session_obj.closed_at = utc_now()
|
||||
return repo.update(session_obj).model_dump(mode="json")
|
||||
@ -17,3 +17,27 @@
|
||||
- v1 的活动计划是静态的:OpenClaw 至多在启动时生成一次,执行期不参与;因此 v1 报告反映的是"预设编排下的周期表现",尚不含 AI 依结果调整的闭环。
|
||||
- v2 把 OpenClaw 放进调度热回路,其可靠性边界是"决策点可用"(非全程存活);决策点不可用时该时段应可跳过/重试而不拖垮整个活动。
|
||||
- 活动是**单对象**聚合;多对象对比通过并行活动 + 跨活动对比实现,不在活动内部横向编排(与 ADR-0001「同场景同版本才可比」一致)。
|
||||
|
||||
---
|
||||
|
||||
## v2 修订(2026-08-03):从"重规划"到"虚拟用户",双轨结构
|
||||
|
||||
原 v2 设想是 OpenClaw 在决策点重排计划条目。经 grilling 重新定位:OpenClaw 的价值不是给固定场景排课表,而是**当虚拟用户本身**——模拟复杂拟真用户行为去打被评对象,产出问题与建议。据此修订为双轨结构:
|
||||
|
||||
1. **周期活动内嵌探索(先行)**:现有活动骨架不变(静态计划照常执行),OpenClaw 常驻代理按巡检结果在窗口内派发探索会话;OpenClaw 停摆只暂停探索,固定计划照常完成。
|
||||
2. **探索活动(后续里程碑)**:独立活动类型,无预设计划,OpenClaw 全权接管评测任务。
|
||||
|
||||
两轨共用底座:探索会话对象、种子配置、体验记录、judge 复核、平台护栏。
|
||||
|
||||
**关键决策**(完整清单见 grilling 记录):
|
||||
|
||||
- **平台耐久底座 + OpenClaw 常驻代理循环**(原"OpenClaw 全程自主驱动"仍被否,但修正为:常驻的是代理进程/循环而非会话;耐久性归平台,自主性归代理)。可靠性边界沿用原文:"决策点可用",不可用时跳过不拖垮活动。
|
||||
- **探索会话是独立实体**,不并入 Run——Run 绑定场景与规则判定,探索会话两者皆无,合并会污染 ADR-0001/0002 的可比性与通过率口径。
|
||||
- **判定 = 体验判定为主 + judge 抽样复核**:意图-结果闭环只有行动者自己能给;judge 补客观质量维度,两条证据链在分析层汇合。
|
||||
- **种子集(种子人设 × 种子目标)是探索式评测的可比性单位**,活动级配置、与 `plan` 同构;代理可在预算内有限衍生。
|
||||
- **巡检载体**:全局一个 OpenClaw heartbeat/cron 常驻作业 + 平台巡检 API(OpenClaw 调度持久化在 SQLite、重启不丢、连续失败自动禁用)。
|
||||
- **护栏平台硬执行**:超预算 API 直接拒绝(默认 ≤8 会话/窗口、≤12 轮/会话、≥30min 间隔)。
|
||||
- **档位**:正式线(time_scale == 1)自动,加速线仅手动——时间压缩与拟真本质冲突。
|
||||
- **产出接入**:报告新增"探索发现"维度 + 喂 v0.7 分析 + Markdown 导出;周期对比暂不扩展,待口径稳定。
|
||||
|
||||
词汇表更新见 `CONTEXT.md`「探索式评测」章节。
|
||||
|
||||
@ -313,6 +313,17 @@ export interface CampaignPlanEntry {
|
||||
count: number
|
||||
}
|
||||
|
||||
export interface ExplorationSeeds {
|
||||
personas: string[]
|
||||
goals: string[]
|
||||
}
|
||||
|
||||
export interface ExplorationBudgetConfig {
|
||||
max_sessions?: number | null
|
||||
max_turns?: number | null
|
||||
min_interval_seconds?: number | null
|
||||
}
|
||||
|
||||
export interface CampaignProgress {
|
||||
current_offset_seconds: number
|
||||
spawned_runs: number
|
||||
@ -337,6 +348,8 @@ export interface Campaign {
|
||||
completed_at: string | null
|
||||
summary: Record<string, unknown> | null
|
||||
analysis_model_config_id: string | null
|
||||
exploration_seeds: ExplorationSeeds | null
|
||||
exploration_budget: ExplorationBudgetConfig | null
|
||||
progress?: CampaignProgress
|
||||
}
|
||||
|
||||
@ -427,6 +440,8 @@ export interface CreateCampaignPayload {
|
||||
time_scale: number
|
||||
plan: CampaignPlanEntry[]
|
||||
analysis_model_config_id?: string | null
|
||||
exploration_seeds?: ExplorationSeeds | null
|
||||
exploration_budget?: ExplorationBudgetConfig | null
|
||||
}
|
||||
|
||||
// ── Period comparison (v0.8) ────────────────────────────────────
|
||||
|
||||
@ -83,6 +83,38 @@ function SectionTitle({ children }: { children: ReactNode }) {
|
||||
return <div style={{ fontWeight: 600, fontSize: 13, margin: '16px 0 12px' }}>{children}</div>
|
||||
}
|
||||
|
||||
function SeedListText({ name, label, placeholder, addLabel }: {
|
||||
name: string
|
||||
label: string
|
||||
placeholder: string
|
||||
addLabel: string
|
||||
}) {
|
||||
return (
|
||||
<>
|
||||
<div style={{ marginBottom: 6, fontSize: 12, color: colors.textSecondary }}>{label}</div>
|
||||
<Form.List name={name}>
|
||||
{(fields, { add, remove }) => (
|
||||
<>
|
||||
{fields.map((field) => (
|
||||
<div key={field.key} style={{ display: 'flex', gap: 8, marginBottom: 8 }}>
|
||||
<Form.Item name={field.name} style={{ flex: 1, marginBottom: 0 }}>
|
||||
<Input placeholder={placeholder} />
|
||||
</Form.Item>
|
||||
<div style={{ width: 16, lineHeight: '32px' }}>
|
||||
<MinusCircleOutlined onClick={() => remove(field.name)} style={{ color: colors.textMuted }} />
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
<Button type="dashed" size="small" onClick={() => add('')} block icon={<PlusOutlined />}>
|
||||
{addLabel}
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
</Form.List>
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
function fmtWindow(seconds: number): string {
|
||||
if (seconds % 3600 === 0) return `${seconds / 3600}h`
|
||||
if (seconds % 60 === 0) return `${seconds / 60}m`
|
||||
@ -181,6 +213,9 @@ export default function CampaignsPage() {
|
||||
realtime: false, target_value: 60, target_unit: 60,
|
||||
plan: [{ scenario_id: undefined, offset_hours: 0, count: 1 }],
|
||||
analysis_model_config_id: null,
|
||||
seed_personas: [], seed_goals: [],
|
||||
exploration_max_sessions: undefined, exploration_max_turns: undefined,
|
||||
exploration_min_interval_minutes: undefined,
|
||||
})
|
||||
setCreateOpen(true)
|
||||
}
|
||||
@ -193,6 +228,16 @@ export default function CampaignsPage() {
|
||||
)
|
||||
setSubmitting(true)
|
||||
try {
|
||||
const personas = ((values.seed_personas as string[] | undefined) ?? [])
|
||||
.map((s) => s?.trim()).filter(Boolean) as string[]
|
||||
const goals = ((values.seed_goals as string[] | undefined) ?? [])
|
||||
.map((s) => s?.trim()).filter(Boolean) as string[]
|
||||
const budget: Record<string, number> = {}
|
||||
if (values.exploration_max_sessions != null) budget.max_sessions = values.exploration_max_sessions
|
||||
if (values.exploration_max_turns != null) budget.max_turns = values.exploration_max_turns
|
||||
if (values.exploration_min_interval_minutes != null) {
|
||||
budget.min_interval_seconds = values.exploration_min_interval_minutes * 60
|
||||
}
|
||||
await campaignsApi.create({
|
||||
name: values.name,
|
||||
target_id: values.target_id,
|
||||
@ -204,6 +249,8 @@ export default function CampaignsPage() {
|
||||
count: e.count ?? 1,
|
||||
})),
|
||||
analysis_model_config_id: (values.analysis_model_config_id as string | null) ?? null,
|
||||
exploration_seeds: personas.length || goals.length ? { personas, goals } : null,
|
||||
exploration_budget: Object.keys(budget).length ? budget : null,
|
||||
})
|
||||
message.success('评估活动已创建并开始调度')
|
||||
setCreateOpen(false)
|
||||
@ -897,6 +944,35 @@ export default function CampaignsPage() {
|
||||
</>
|
||||
)}
|
||||
</Form.List>
|
||||
|
||||
<SectionTitle>
|
||||
探索式评测(虚拟用户){' '}
|
||||
<Tooltip title="AI 助手以种子人设 × 种子目标自主与被评对象对话;种子留空则该活动不参与探索">
|
||||
<QuestionCircleOutlined style={{ color: colors.textMuted }} />
|
||||
</Tooltip>
|
||||
</SectionTitle>
|
||||
<Row gutter={16}>
|
||||
<Col span={12}>
|
||||
<SeedListText name="seed_personas" label="种子人设" placeholder="如:急性子的缴费用户" addLabel="添加人设" />
|
||||
</Col>
|
||||
<Col span={12}>
|
||||
<SeedListText name="seed_goals" label="种子目标" placeholder="如:查询本月账单并完成缴费" addLabel="添加目标" />
|
||||
</Col>
|
||||
</Row>
|
||||
<div style={{ marginTop: 12, marginBottom: 6, fontSize: 12, color: colors.textSecondary }}>
|
||||
探索预算覆盖(留空使用平台默认:8 会话/窗口 · 12 轮/会话 · 30 分钟间隔)
|
||||
</div>
|
||||
<div style={{ display: 'flex', gap: 8 }}>
|
||||
<Form.Item name="exploration_max_sessions" style={{ flex: 1, marginBottom: 0 }}>
|
||||
<InputNumber min={1} placeholder="默认 8" addonAfter="会话/窗口" style={{ width: '100%' }} />
|
||||
</Form.Item>
|
||||
<Form.Item name="exploration_max_turns" style={{ flex: 1, marginBottom: 0 }}>
|
||||
<InputNumber min={1} placeholder="默认 12" addonAfter="轮/会话" style={{ width: '100%' }} />
|
||||
</Form.Item>
|
||||
<Form.Item name="exploration_min_interval_minutes" style={{ flex: 1, marginBottom: 0 }}>
|
||||
<InputNumber min={1} placeholder="默认 30" addonAfter="分钟间隔" style={{ width: '100%' }} />
|
||||
</Form.Item>
|
||||
</div>
|
||||
</Col>
|
||||
</Row>
|
||||
</Form>
|
||||
|
||||
69
migrations/versions/0e4a7c91d2b3_add_exploration_tables.py
Normal file
69
migrations/versions/0e4a7c91d2b3_add_exploration_tables.py
Normal file
@ -0,0 +1,69 @@
|
||||
"""add exploration_sessions and exploration_messages tables
|
||||
|
||||
Revision ID: 0e4a7c91d2b3
|
||||
Revises: f2a9b7c34d18
|
||||
Create Date: 2026-08-03
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
import sqlmodel # noqa: F401
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0e4a7c91d2b3"
|
||||
down_revision: Union[str, Sequence[str], None] = "f2a9b7c34d18"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"exploration_sessions",
|
||||
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("campaign_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("target_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("persona", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("goal", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("seed_ref", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
|
||||
sa.Column("status", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("triggered_by", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("experience", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
|
||||
sa.Column("judge_review", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
|
||||
sa.Column("turn_count", sa.Integer(), nullable=False),
|
||||
sa.Column("error", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(), nullable=True),
|
||||
sa.Column("closed_at", sa.DateTime(), nullable=True),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.ForeignKeyConstraint(["campaign_id"], ["campaigns.id"]),
|
||||
sa.ForeignKeyConstraint(["target_id"], ["eval_targets.id"]),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_exploration_sessions_campaign_id",
|
||||
"exploration_sessions",
|
||||
["campaign_id"],
|
||||
)
|
||||
op.create_table(
|
||||
"exploration_messages",
|
||||
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("session_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("round_index", sa.Integer(), nullable=False),
|
||||
sa.Column("role", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("content", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
|
||||
sa.Column("latency_ms", sa.Integer(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(), nullable=True),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.ForeignKeyConstraint(["session_id"], ["exploration_sessions.id"]),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_exploration_messages_session_id",
|
||||
"exploration_messages",
|
||||
["session_id"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_exploration_messages_session_id")
|
||||
op.drop_table("exploration_messages")
|
||||
op.drop_index("ix_exploration_sessions_campaign_id")
|
||||
op.drop_table("exploration_sessions")
|
||||
@ -0,0 +1,29 @@
|
||||
"""add exploration seeds/budget config to campaigns
|
||||
|
||||
Revision ID: b3c7d9e1f5a2
|
||||
Revises: 0e4a7c91d2b3
|
||||
Create Date: 2026-08-03
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
import sqlmodel # noqa: F401
|
||||
from alembic import op
|
||||
|
||||
revision: str = "b3c7d9e1f5a2"
|
||||
down_revision: Union[str, Sequence[str], None] = "0e4a7c91d2b3"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("campaigns") as batch_op:
|
||||
batch_op.add_column(sa.Column("exploration_seeds", sqlmodel.sql.sqltypes.AutoString(), nullable=True))
|
||||
batch_op.add_column(sa.Column("exploration_budget", sqlmodel.sql.sqltypes.AutoString(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("campaigns") as batch_op:
|
||||
batch_op.drop_column("exploration_budget")
|
||||
batch_op.drop_column("exploration_seeds")
|
||||
@ -0,0 +1,26 @@
|
||||
"""add campaigns.last_patrolled_at watermark
|
||||
|
||||
Revision ID: c5d8f0a2e4b7
|
||||
Revises: b3c7d9e1f5a2
|
||||
Create Date: 2026-08-03
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "c5d8f0a2e4b7"
|
||||
down_revision: Union[str, Sequence[str], None] = "b3c7d9e1f5a2"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("campaigns") as batch_op:
|
||||
batch_op.add_column(sa.Column("last_patrolled_at", sa.DateTime(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("campaigns") as batch_op:
|
||||
batch_op.drop_column("last_patrolled_at")
|
||||
@ -35,6 +35,8 @@ def db_session(tmp_db_path: Path) -> Session:
|
||||
EvalResultDB,
|
||||
EvalRunDB,
|
||||
EvalTargetDB,
|
||||
ExplorationMessageDB,
|
||||
ExplorationSessionDB,
|
||||
ModelConfigDB,
|
||||
ScenarioDB,
|
||||
ScenarioModelBindingDB,
|
||||
@ -49,6 +51,7 @@ def db_session(tmp_db_path: Path) -> Session:
|
||||
|
||||
# Sanity check: verify the scenarios table has all expected columns.
|
||||
from sqlalchemy import inspect as sa_inspect
|
||||
|
||||
cols = [c["name"] for c in sa_inspect(engine).get_columns("scenarios")]
|
||||
assert "llm_config" in cols, f"scenarios table missing llm_config; cols={cols}"
|
||||
|
||||
|
||||
@ -377,3 +377,64 @@ async def test_create_campaign_without_override_stores_null(client, seeded_db):
|
||||
async def test_create_campaign_invalid_analysis_model_400(client, seeded_db):
|
||||
resp = await client.post("/api/campaigns", json=_valid_payload(analysis_model_config_id="nope"))
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
async def test_create_campaign_with_exploration_config(client, seeded_db):
|
||||
payload = _valid_payload(
|
||||
exploration_seeds={"personas": ["急性子用户", "谨慎的老年用户"], "goals": ["查询账单并缴费", "修改收货地址"]},
|
||||
exploration_budget={"max_sessions": 4, "max_turns": 6, "min_interval_seconds": 3600},
|
||||
)
|
||||
resp = await client.post("/api/campaigns", json=payload)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["exploration_seeds"] == {
|
||||
"personas": ["急性子用户", "谨慎的老年用户"],
|
||||
"goals": ["查询账单并缴费", "修改收货地址"],
|
||||
}
|
||||
assert body["exploration_budget"] == {"max_sessions": 4, "max_turns": 6, "min_interval_seconds": 3600}
|
||||
|
||||
got = (await client.get(f"/api/campaigns/{body['id']}")).json()
|
||||
assert got["exploration_seeds"]["personas"] == ["急性子用户", "谨慎的老年用户"]
|
||||
assert got["exploration_budget"]["max_turns"] == 6
|
||||
|
||||
|
||||
async def test_create_campaign_without_exploration_config(client, seeded_db):
|
||||
body = (await client.post("/api/campaigns", json=_valid_payload())).json()
|
||||
assert body["exploration_seeds"] is None
|
||||
assert body["exploration_budget"] is None
|
||||
|
||||
|
||||
async def test_empty_seeds_campaign_opts_out_of_exploration(client, seeded_db):
|
||||
payload = _valid_payload(exploration_seeds={"personas": [], "goals": []})
|
||||
body = (await client.post("/api/campaigns", json=payload)).json()
|
||||
assert body["exploration_seeds"] is None
|
||||
|
||||
|
||||
def test_exploration_config_migration_on_existing_db(tmp_path, monkeypatch):
|
||||
"""The two campaign config columns apply on a DB at the previous head."""
|
||||
from pathlib import Path
|
||||
|
||||
from agenteval.storage import db as db_module
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import create_engine, inspect, text
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
database_url = f"sqlite:///{tmp_path / 'campaign_config.db'}"
|
||||
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
|
||||
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
|
||||
|
||||
SQLModel.metadata.create_all(create_engine(database_url))
|
||||
with create_engine(database_url).begin() as connection:
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
|
||||
|
||||
command.stamp(config, "0e4a7c91d2b3")
|
||||
command.upgrade(config, "head")
|
||||
|
||||
cols = {c["name"] for c in inspect(create_engine(database_url)).get_columns("campaigns")}
|
||||
assert {"exploration_seeds", "exploration_budget"} <= cols
|
||||
|
||||
391
tests/integration/test_exploration_api.py
Normal file
391
tests/integration/test_exploration_api.py
Normal file
@ -0,0 +1,391 @@
|
||||
"""Integration tests for exploration session lifecycle + platform guardrails (v0.9 票据 01).
|
||||
|
||||
Uses ``httpx.AsyncClient`` with ``app=`` to drive the FastAPI app in-process.
|
||||
Channel I/O is stubbed with MockChannel; tests cover the full lifecycle
|
||||
(create → message → close), the three budget guardrails (409), trigger-source
|
||||
gating per line tier, and experience-record normalization.
|
||||
"""
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from agenteval.models import Campaign, ChannelType, EvalTarget, PlatformType, TargetStatus
|
||||
from agenteval.storage.db import ExplorationSessionDB, utc_now
|
||||
from agenteval.storage.repository import CampaignRepository, ExplorationSessionRepository, TargetRepository
|
||||
from agenteval.web.app import app
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
|
||||
def _make_campaign(campaign_id: str, *, time_scale: float = 1.0, status: str = "running") -> Campaign:
|
||||
return Campaign(
|
||||
id=campaign_id,
|
||||
name=f"campaign-{campaign_id}",
|
||||
target_id="t-1",
|
||||
window_seconds=86400,
|
||||
time_scale=time_scale,
|
||||
plan=[{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
|
||||
status=status,
|
||||
started_at=utc_now(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_db(db_session, monkeypatch):
|
||||
"""Patch get_session/get_db to the test session and seed target + campaign."""
|
||||
from agenteval.storage import db as db_module
|
||||
from agenteval.storage import repository as repo_module
|
||||
from agenteval.web import app as app_module
|
||||
|
||||
monkeypatch.setattr(app_module, "init_db", lambda: None)
|
||||
|
||||
def _test_get_session():
|
||||
return db_session
|
||||
|
||||
monkeypatch.setattr(db_module, "get_session", _test_get_session)
|
||||
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
|
||||
|
||||
from agenteval.web.deps import get_db
|
||||
|
||||
def _test_get_db():
|
||||
try:
|
||||
yield db_session
|
||||
finally:
|
||||
pass
|
||||
|
||||
app.dependency_overrides[get_db] = _test_get_db
|
||||
|
||||
target = EvalTarget(
|
||||
id="t-1",
|
||||
name="mock-target",
|
||||
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
|
||||
channel_type=ChannelType.TUTU_API,
|
||||
channel_config={"base_url": "http://mock", "token": "x"},
|
||||
status=TargetStatus.ACTIVE,
|
||||
)
|
||||
TargetRepository(db_session).create(target)
|
||||
CampaignRepository(db_session).create(_make_campaign("c-1"))
|
||||
|
||||
yield db_session
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def _stub_channel_factory(monkeypatch, channel) -> None:
|
||||
"""Point the exploration router's ChannelFactory at a test channel."""
|
||||
from agenteval.web.routers import exploration as exploration_module
|
||||
|
||||
class _StubFactory:
|
||||
@staticmethod
|
||||
def create(target):
|
||||
return channel
|
||||
|
||||
monkeypatch.setattr(exploration_module, "ChannelFactory", _StubFactory)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def mock_channel(monkeypatch):
|
||||
"""Stub ChannelFactory in the exploration router with a MockChannel."""
|
||||
from tests.unit.mock_channel import MockChannel
|
||||
|
||||
channel = MockChannel(reply_text="您好,请问有什么可以帮您?")
|
||||
_stub_channel_factory(monkeypatch, channel)
|
||||
return channel
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
async def client():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _session_payload(**overrides) -> dict:
|
||||
payload = {
|
||||
"campaign_id": "c-1",
|
||||
"persona": {"name": "急性子用户", "traits": ["急躁", "目标导向"]},
|
||||
"goal": "查询本月账单并完成缴费",
|
||||
"triggered_by": "auto",
|
||||
}
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
|
||||
def _rewind_latest_session(db_session, minutes: int = 31) -> None:
|
||||
"""Move the newest session's created_at back so the interval guardrail passes."""
|
||||
repo = ExplorationSessionRepository(db_session)
|
||||
sessions = repo.list_by_campaign("c-1")
|
||||
latest = max(sessions, key=lambda s: s.created_at)
|
||||
row = db_session.get(ExplorationSessionDB, latest.id)
|
||||
row.created_at = utc_now() - timedelta(minutes=minutes)
|
||||
db_session.add(row)
|
||||
db_session.commit()
|
||||
|
||||
|
||||
async def _create_session(client, **overrides):
|
||||
return await client.post("/api/exploration/sessions", json=_session_payload(**overrides))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- lifecycle
|
||||
|
||||
|
||||
async def test_full_lifecycle_create_message_close(seeded_db, mock_channel, client):
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 200, resp.text
|
||||
session = resp.json()
|
||||
assert session["status"] == "running"
|
||||
assert session["campaign_id"] == "c-1"
|
||||
assert session["target_id"] == "t-1"
|
||||
assert session["persona"]["name"] == "急性子用户"
|
||||
session_id = session["id"]
|
||||
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": "我要查这个月的账单"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["reply"] == "您好,请问有什么可以帮您?"
|
||||
assert isinstance(body["latency_ms"], int)
|
||||
assert body["turn_count"] == 1
|
||||
|
||||
repo = ExplorationSessionRepository(seeded_db)
|
||||
assert repo.get(session_id).turn_count == 1
|
||||
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/close",
|
||||
json={
|
||||
"experience": {
|
||||
"goal_achieved": True,
|
||||
"blockers": [],
|
||||
"misled": [],
|
||||
"emotion": "positive",
|
||||
"notes": "顺利完成",
|
||||
}
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
closed = resp.json()
|
||||
assert closed["status"] == "completed"
|
||||
assert closed["experience"]["goal_achieved"] is True
|
||||
assert closed["experience"]["emotion"] == "positive"
|
||||
assert closed["closed_at"] is not None
|
||||
|
||||
|
||||
async def test_create_requires_existing_running_campaign(seeded_db, client):
|
||||
resp = await _create_session(client, campaign_id="nope")
|
||||
assert resp.status_code == 404
|
||||
|
||||
CampaignRepository(seeded_db).create(_make_campaign("c-done", status="completed"))
|
||||
resp = await _create_session(client, campaign_id="c-done")
|
||||
assert resp.status_code == 409
|
||||
assert "进行中" in resp.json()["detail"]
|
||||
|
||||
|
||||
async def test_accelerated_line_accepts_manual_only(seeded_db, mock_channel, client):
|
||||
CampaignRepository(seeded_db).create(_make_campaign("c-fast", time_scale=24.0))
|
||||
|
||||
resp = await _create_session(client, campaign_id="c-fast", triggered_by="auto")
|
||||
assert resp.status_code == 409
|
||||
assert "手动" in resp.json()["detail"]
|
||||
|
||||
resp = await _create_session(client, campaign_id="c-fast", triggered_by="manual")
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
async def test_session_budget_guardrail(seeded_db, mock_channel, client):
|
||||
for _ in range(8):
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 200, resp.text
|
||||
_rewind_latest_session(seeded_db)
|
||||
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 409
|
||||
assert "预算" in resp.json()["detail"]
|
||||
|
||||
|
||||
async def test_session_interval_guardrail(seeded_db, mock_channel, client):
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 200
|
||||
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 409
|
||||
assert "间隔" in resp.json()["detail"]
|
||||
|
||||
_rewind_latest_session(seeded_db)
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
async def test_campaign_budget_override_limits_sessions(seeded_db, mock_channel, client):
|
||||
"""活动级预算覆盖生效(票据 02):max_sessions=1 时第二个会话被拒。"""
|
||||
campaign = CampaignRepository(seeded_db).get("c-1")
|
||||
campaign.exploration_budget = {"max_sessions": 1}
|
||||
CampaignRepository(seeded_db).update(campaign)
|
||||
|
||||
assert (await _create_session(client)).status_code == 200
|
||||
_rewind_latest_session(seeded_db)
|
||||
resp = await _create_session(client)
|
||||
assert resp.status_code == 409
|
||||
assert "预算" in resp.json()["detail"]
|
||||
|
||||
|
||||
async def test_turn_budget_guardrail(seeded_db, mock_channel, client):
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
for i in range(12):
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": f"第 {i + 1} 轮问题"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": "第 13 轮问题"},
|
||||
)
|
||||
assert resp.status_code == 409
|
||||
assert "轮数" in resp.json()["detail"]
|
||||
|
||||
|
||||
async def test_message_rejected_when_session_not_running(seeded_db, mock_channel, client):
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/close",
|
||||
json={"experience": {"goal_achieved": False}},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": "还在吗?"},
|
||||
)
|
||||
assert resp.status_code == 409
|
||||
assert "进行中" in resp.json()["detail"]
|
||||
|
||||
|
||||
async def test_message_unknown_session_returns_404(seeded_db, mock_channel, client):
|
||||
resp = await client.post("/api/exploration/sessions/nope/messages", json={"content": "hi"})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
async def test_channel_failure_returns_502_without_consuming_turn(seeded_db, monkeypatch, client):
|
||||
from tests.unit.mock_channel import MockChannel
|
||||
|
||||
channel = MockChannel(send_ok=False)
|
||||
_stub_channel_factory(monkeypatch, channel)
|
||||
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": "你好"},
|
||||
)
|
||||
assert resp.status_code == 502
|
||||
assert ExplorationSessionRepository(seeded_db).get(session_id).turn_count == 0
|
||||
|
||||
|
||||
async def test_poll_timeout_consumes_turn_budget(seeded_db, monkeypatch, client):
|
||||
"""消息已送达但等不到回复:账本仍计一轮(超时不可绕过轮数预算)。"""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from agenteval.web.routers import exploration as exploration_module
|
||||
|
||||
from tests.unit.mock_channel import MockChannel
|
||||
|
||||
channel = MockChannel(missing_reply=True)
|
||||
_stub_channel_factory(monkeypatch, channel)
|
||||
monkeypatch.setattr(exploration_module, "get_settings", lambda: SimpleNamespace(poll_reply_timeout=0.05))
|
||||
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/messages",
|
||||
json={"content": "有人在吗"},
|
||||
)
|
||||
assert resp.status_code == 502
|
||||
assert ExplorationSessionRepository(seeded_db).get(session_id).turn_count == 1
|
||||
|
||||
|
||||
async def test_close_normalizes_experience(seeded_db, mock_channel, client):
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
resp = await client.post(
|
||||
f"/api/exploration/sessions/{session_id}/close",
|
||||
json={
|
||||
"experience": {
|
||||
"blockers": [42, {"not": "a string"}],
|
||||
"misled": "不是列表",
|
||||
"emotion": "暴怒!!!",
|
||||
}
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
experience = resp.json()["experience"]
|
||||
assert experience["goal_achieved"] is False
|
||||
assert experience["blockers"] == ["42"]
|
||||
assert experience["misled"] == []
|
||||
assert experience["emotion"] == "neutral"
|
||||
|
||||
|
||||
async def test_close_twice_rejected(seeded_db, mock_channel, client):
|
||||
session_id = (await _create_session(client)).json()["id"]
|
||||
payload = {"experience": {"goal_achieved": True}}
|
||||
assert (await client.post(f"/api/exploration/sessions/{session_id}/close", json=payload)).status_code == 200
|
||||
resp = await client.post(f"/api/exploration/sessions/{session_id}/close", json=payload)
|
||||
assert resp.status_code == 409
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- migration
|
||||
|
||||
|
||||
def test_exploration_migration_on_existing_db(tmp_path, monkeypatch):
|
||||
"""Alembic migration applies on an existing DB at the previous head.
|
||||
|
||||
Brand-new DBs take the create_all path (exercised by every test above via
|
||||
the db_session fixture, which creates the new tables from metadata).
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from agenteval.storage import db as db_module
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import create_engine, inspect
|
||||
|
||||
database_url = f"sqlite:///{tmp_path / 'exploration.db'}"
|
||||
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
|
||||
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
|
||||
|
||||
# 既有库先例:基线迁移假设 create_all 建好的表已存在,先 stamp 基线前状态
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
engine = create_engine(database_url)
|
||||
SQLModel.metadata.create_all(engine)
|
||||
engine.dispose()
|
||||
with create_engine(database_url).begin() as connection:
|
||||
from sqlalchemy import text
|
||||
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
|
||||
|
||||
command.stamp(config, "f2a9b7c34d18")
|
||||
command.upgrade(config, "head")
|
||||
|
||||
inspector = inspect(create_engine(database_url))
|
||||
tables = set(inspector.get_table_names())
|
||||
assert "exploration_sessions" in tables
|
||||
assert "exploration_messages" in tables
|
||||
session_cols = {c["name"] for c in inspector.get_columns("exploration_sessions")}
|
||||
assert {
|
||||
"campaign_id",
|
||||
"target_id",
|
||||
"persona",
|
||||
"goal",
|
||||
"seed_ref",
|
||||
"status",
|
||||
"triggered_by",
|
||||
"experience",
|
||||
"judge_review",
|
||||
"turn_count",
|
||||
"closed_at",
|
||||
} <= session_cols
|
||||
message_cols = {c["name"] for c in inspector.get_columns("exploration_messages")}
|
||||
assert {"session_id", "round_index", "role", "content", "latency_ms"} <= message_cols
|
||||
203
tests/integration/test_exploration_patrol_api.py
Normal file
203
tests/integration/test_exploration_patrol_api.py
Normal file
@ -0,0 +1,203 @@
|
||||
"""Integration tests for the exploration patrol API (v0.9 票据 03).
|
||||
|
||||
One stateless call returns every running production-line campaign that
|
||||
participates in exploration, the new results since the last patrol watermark
|
||||
(reusing the campaign report aggregation), and the remaining exploration
|
||||
budget. The watermark advances after each call so subsequent calls only
|
||||
report increments.
|
||||
"""
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from agenteval.models import Campaign, EvalRun, RunStatus, RunSummary
|
||||
from agenteval.storage.db import utc_now
|
||||
from agenteval.storage.repository import CampaignRepository, RunRepository
|
||||
from agenteval.web.app import app
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
SEEDS = {"personas": ["急性子用户"], "goals": ["查询账单并缴费"]}
|
||||
|
||||
|
||||
def _make_campaign(campaign_id: str, *, time_scale: float = 1.0, status: str = "running", seeds=SEEDS) -> Campaign:
|
||||
return Campaign(
|
||||
id=campaign_id,
|
||||
name=f"campaign-{campaign_id}",
|
||||
target_id="t-1",
|
||||
window_seconds=86400,
|
||||
time_scale=time_scale,
|
||||
plan=[{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
|
||||
status=status,
|
||||
started_at=utc_now() - timedelta(hours=2),
|
||||
exploration_seeds=seeds,
|
||||
)
|
||||
|
||||
|
||||
def _seed_run(db_session, run_id: str, campaign_id: str, *, pass_rate: float, completed_at) -> None:
|
||||
RunRepository(db_session).create(EvalRun(
|
||||
id=run_id, target_id="t-1", scenario_id="s-1", campaign_id=campaign_id,
|
||||
status=RunStatus.COMPLETED, started_at=completed_at - timedelta(minutes=5),
|
||||
completed_at=completed_at,
|
||||
summary=RunSummary(total_cases=2, pass_rate=pass_rate, avg_latency_ms=120.0),
|
||||
))
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_db(db_session, monkeypatch):
|
||||
from agenteval.models import ChannelType, EvalTarget, PlatformType, TargetStatus
|
||||
from agenteval.storage import db as db_module
|
||||
from agenteval.storage import repository as repo_module
|
||||
from agenteval.storage.repository import TargetRepository
|
||||
from agenteval.web import app as app_module
|
||||
|
||||
monkeypatch.setattr(app_module, "init_db", lambda: None)
|
||||
|
||||
def _test_get_session():
|
||||
return db_session
|
||||
|
||||
monkeypatch.setattr(db_module, "get_session", _test_get_session)
|
||||
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
|
||||
|
||||
from agenteval.web.deps import get_db
|
||||
|
||||
def _test_get_db():
|
||||
try:
|
||||
yield db_session
|
||||
finally:
|
||||
pass
|
||||
|
||||
app.dependency_overrides[get_db] = _test_get_db
|
||||
|
||||
TargetRepository(db_session).create(EvalTarget(
|
||||
id="t-1", name="mock-target",
|
||||
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
|
||||
channel_type=ChannelType.TUTU_API,
|
||||
channel_config={"base_url": "http://mock", "token": "x"},
|
||||
status=TargetStatus.ACTIVE,
|
||||
))
|
||||
repo = CampaignRepository(db_session)
|
||||
repo.create(_make_campaign("c-prod")) # 正式线,参与探索
|
||||
repo.create(_make_campaign("c-fast", time_scale=24.0)) # 加速线 → 不巡检
|
||||
repo.create(_make_campaign("c-noseed", seeds=None)) # 无种子集 → 不巡检
|
||||
repo.create(_make_campaign("c-done", status="completed")) # 终态 → 不巡检
|
||||
|
||||
_seed_run(db_session, "r-1", "c-prod", pass_rate=1.0, completed_at=utc_now() - timedelta(hours=1))
|
||||
_seed_run(db_session, "r-2", "c-prod", pass_rate=0.5, completed_at=utc_now() - timedelta(minutes=30))
|
||||
|
||||
yield db_session
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
async def client():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
async def _patrol(client) -> dict:
|
||||
resp = await client.get("/api/exploration/patrol")
|
||||
assert resp.status_code == 200, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
async def test_patrol_filters_to_running_production_seeded_campaigns(client, seeded_db):
|
||||
body = await _patrol(client)
|
||||
ids = [c["campaign_id"] for c in body["campaigns"]]
|
||||
assert ids == ["c-prod"]
|
||||
|
||||
|
||||
async def test_patrol_entry_content(client, seeded_db):
|
||||
body = await _patrol(client)
|
||||
entry = body["campaigns"][0]
|
||||
assert entry["campaign_name"] == "campaign-c-prod"
|
||||
assert entry["target_id"] == "t-1"
|
||||
assert entry["target_name"] == "mock-target"
|
||||
assert entry["last_patrolled_at"] is None # 首次巡检无水位
|
||||
|
||||
new_results = entry["new_results"]
|
||||
assert new_results is not None
|
||||
assert new_results["summary"]["total_runs"] == 2
|
||||
assert new_results["summary"]["overall_pass_rate"] == 0.75
|
||||
assert new_results["capability_summary"][0]["scenario_id"] == "s-1"
|
||||
|
||||
budget = entry["budget"]
|
||||
assert budget["max_sessions"] == 8
|
||||
assert budget["sessions_used"] == 0
|
||||
assert budget["remaining_sessions"] == 8
|
||||
assert budget["max_turns"] == 12
|
||||
assert budget["min_interval_seconds"] == 30 * 60
|
||||
assert budget["seconds_since_last_session"] is None
|
||||
|
||||
|
||||
async def test_patrol_watermark_advances_and_reports_increments(client, seeded_db):
|
||||
await _patrol(client)
|
||||
campaign = CampaignRepository(seeded_db).get("c-prod")
|
||||
assert campaign.last_patrolled_at is not None
|
||||
|
||||
# 第二次巡检:无新完成的子运行 → 增量为空
|
||||
body = await _patrol(client)
|
||||
entry = body["campaigns"][0]
|
||||
assert entry["new_results"] is None
|
||||
|
||||
# 水位之后新完成一个子运行 → 第三次巡检只报这一个
|
||||
_seed_run(seeded_db, "r-3", "c-prod", pass_rate=0.0, completed_at=utc_now())
|
||||
body = await _patrol(client)
|
||||
entry = body["campaigns"][0]
|
||||
assert entry["new_results"]["summary"]["total_runs"] == 1
|
||||
assert entry["new_results"]["summary"]["overall_pass_rate"] == 0.0
|
||||
|
||||
|
||||
async def test_patrol_budget_reflects_existing_sessions(client, seeded_db):
|
||||
from agenteval.exploration.models import ExplorationSession
|
||||
from agenteval.storage.repository import ExplorationSessionRepository
|
||||
|
||||
ExplorationSessionRepository(seeded_db).create(ExplorationSession(
|
||||
campaign_id="c-prod", target_id="t-1", goal="查询账单", persona={"name": "x"},
|
||||
))
|
||||
body = await _patrol(client)
|
||||
budget = body["campaigns"][0]["budget"]
|
||||
assert budget["sessions_used"] == 1
|
||||
assert budget["remaining_sessions"] == 7
|
||||
assert budget["seconds_since_last_session"] is not None
|
||||
|
||||
|
||||
async def test_patrol_budget_honours_campaign_override(client, seeded_db):
|
||||
campaign = CampaignRepository(seeded_db).get("c-prod")
|
||||
campaign.exploration_budget = {"max_sessions": 3}
|
||||
CampaignRepository(seeded_db).update(campaign)
|
||||
|
||||
body = await _patrol(client)
|
||||
budget = body["campaigns"][0]["budget"]
|
||||
assert budget["max_sessions"] == 3
|
||||
assert budget["remaining_sessions"] == 3
|
||||
|
||||
|
||||
async def test_patrol_migration_column_on_existing_db(tmp_path, monkeypatch):
|
||||
"""last_patrolled_at applies on a DB at the previous head."""
|
||||
from pathlib import Path
|
||||
|
||||
from agenteval.storage import db as db_module
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import create_engine, inspect, text
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
database_url = f"sqlite:///{tmp_path / 'patrol.db'}"
|
||||
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
|
||||
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
|
||||
|
||||
SQLModel.metadata.create_all(create_engine(database_url))
|
||||
with create_engine(database_url).begin() as connection:
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_sessions"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS exploration_messages"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_seeds"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN exploration_budget"))
|
||||
connection.execute(text("ALTER TABLE campaigns DROP COLUMN last_patrolled_at"))
|
||||
connection.execute(text("DROP TABLE IF EXISTS alembic_version"))
|
||||
|
||||
command.stamp(config, "b3c7d9e1f5a2")
|
||||
command.upgrade(config, "head")
|
||||
|
||||
cols = {c["name"] for c in inspect(create_engine(database_url)).get_columns("campaigns")}
|
||||
assert "last_patrolled_at" in cols
|
||||
38
tests/unit/test_exploration_budget.py
Normal file
38
tests/unit/test_exploration_budget.py
Normal file
@ -0,0 +1,38 @@
|
||||
"""Unit tests for exploration budget resolution (platform defaults + campaign override)."""
|
||||
|
||||
from agenteval.exploration.models import resolve_budget
|
||||
from agenteval.models import Campaign
|
||||
|
||||
|
||||
def _campaign(**overrides) -> Campaign:
|
||||
base = {
|
||||
"name": "c",
|
||||
"target_id": "t-1",
|
||||
"window_seconds": 86400,
|
||||
"plan": [{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
|
||||
}
|
||||
base.update(overrides)
|
||||
return Campaign(**base)
|
||||
|
||||
|
||||
def test_platform_defaults_when_no_override():
|
||||
budget = resolve_budget(_campaign())
|
||||
assert budget.max_sessions == 8
|
||||
assert budget.max_turns == 12
|
||||
assert budget.min_interval_seconds == 30 * 60
|
||||
|
||||
|
||||
def test_partial_override_falls_back_to_defaults():
|
||||
campaign = _campaign(exploration_budget={"max_sessions": 3})
|
||||
budget = resolve_budget(campaign)
|
||||
assert budget.max_sessions == 3
|
||||
assert budget.max_turns == 12
|
||||
assert budget.min_interval_seconds == 30 * 60
|
||||
|
||||
|
||||
def test_full_override():
|
||||
campaign = _campaign(
|
||||
exploration_budget={"max_sessions": 2, "max_turns": 5, "min_interval_seconds": 600}
|
||||
)
|
||||
budget = resolve_budget(campaign)
|
||||
assert (budget.max_sessions, budget.max_turns, budget.min_interval_seconds) == (2, 5, 600)
|
||||
Loading…
Reference in New Issue
Block a user