Compare commits
2 Commits
e1491dfc97
...
6cc2efafb6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6cc2efafb6 | ||
|
|
9cdbc41808 |
21
.scratch/v1.0/POST_V1_IMPROVEMENTS.md
Normal file
21
.scratch/v1.0/POST_V1_IMPROVEMENTS.md
Normal file
@ -0,0 +1,21 @@
|
||||
# 智能评估 v1.0 后续优化点
|
||||
|
||||
## 前端体验优化
|
||||
|
||||
### 1. 详情页自动刷新 ✅ 已修复
|
||||
**问题**:详情页在 `planning` 或 `executing` 状态下不会自动刷新,OpenClaw 改变状态后 UI 仍显示旧状态,需要返回列表再进入才能看到更新。
|
||||
|
||||
**修复**:详情页在 `planning`/`executing`/`pending_approval` 状态下每 5 秒静默轮询评估详情(IntelligentEvals.tsx)。
|
||||
|
||||
### 2. 执行期会话列表展示 ✅ 已修复
|
||||
**问题**:详情页在 `executing` 状态下只显示汇总数字(`completed_sessions / session_count`),无法看到具体哪些会话已创建、进行中、已完成。
|
||||
|
||||
**修复**:详情页新增「会话进度」卡片,逐会话展示人设/状态/维度/目标/轮数/创建时间;`executing` 状态下同样每 5 秒轮询会话列表(EvalDetail.tsx)。
|
||||
|
||||
---
|
||||
|
||||
## 记录时间
|
||||
2026-08-05
|
||||
|
||||
## 优先级
|
||||
P2(v1.0 功能已完整,这些是体验优化)
|
||||
17
.scratch/v1.0/issues/01-data-model.md
Normal file
17
.scratch/v1.0/issues/01-data-model.md
Normal file
@ -0,0 +1,17 @@
|
||||
# 01 — 智能评估数据模型 + 迁移
|
||||
|
||||
**What to build:** 创建智能评估的三个数据表(IntelligentEval、IntelligentEvalSession、IntelligentEvalMessage)及对应的 Repository 类,走 Alembic batch mode 迁移。
|
||||
|
||||
**Blocked by:** 无
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `IntelligentEvalDB` 表定义:id, name, target_id, status, goal, seeds(JSON), intent, role_description, plan(JSON), plan_feedback, time_window_hours, report(JSON), created_at, updated_at, started_at, completed_at
|
||||
- [x] `IntelligentEvalSessionDB` 表定义:id, eval_id, target_id, persona(JSON), goal, dimension, status, verdict(JSON), turn_count, created_at, closed_at
|
||||
- [x] `IntelligentEvalMessageDB` 表定义:id, session_id, role, content, latency_ms, created_at
|
||||
- [x] `IntelligentEvalStatus` 枚举:draft, planning, pending_approval, executing, completed, cancelled, failed
|
||||
- [x] `IntelligentEvalSessionStatus` 枚举:running, completed, failed, expired
|
||||
- [x] `IntelligentEvalRepository`:CRUD + list + 状态转换方法
|
||||
- [x] `IntelligentEvalSessionRepository`:CRUD + list_by_eval
|
||||
- [x] `IntelligentEvalMessageRepository`:create + list_by_session
|
||||
- [x] Alembic 迁移(batch mode)通过
|
||||
- [x] `init_db()` 的 `create_all()` 能建出三张表
|
||||
17
.scratch/v1.0/issues/02-lifecycle-api.md
Normal file
17
.scratch/v1.0/issues/02-lifecycle-api.md
Normal file
@ -0,0 +1,17 @@
|
||||
# 02 — 智能评估核心生命周期 API
|
||||
|
||||
**What to build:** 智能评估的创建、规划、审批、取消、列表、详情 API。覆盖状态机 draft → planning → pending_approval → executing / cancelled 的转换。
|
||||
|
||||
**Blocked by:** 01(数据模型就绪)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `POST /api/intelligent-evals`:创建(body = name, target_id, goal, seeds, intent, role_description, time_window_hours),状态 draft → planning
|
||||
- [x] `PUT /api/intelligent-evals/{id}/plan`:OpenClaw 提交粗计划(body = plan JSON),状态 planning → pending_approval
|
||||
- [x] `POST /api/intelligent-evals/{id}/approve`:用户批准,状态 pending_approval → executing,记录 started_at
|
||||
- [x] `POST /api/intelligent-evals/{id}/reject`:用户打回(body = feedback),状态 pending_approval → planning,记录 plan_feedback
|
||||
- [x] `POST /api/intelligent-evals/{id}/cancel`:用户取消,状态 → cancelled(仅 pending_approval / executing 可取消)
|
||||
- [x] `GET /api/intelligent-evals`:列表(含 status、target 名称)
|
||||
- [x] `GET /api/intelligent-evals/{id}`:详情(含完整 plan、进度信息)
|
||||
- [x] 状态机非法转换返回 409
|
||||
- [x] 全部 X-API-Key 鉴权(沿用 `require_api_key`)
|
||||
- [x] 集成测试覆盖:创建 → 规划 → 审批 → 执行完整路径;打回 → 重新规划路径;取消路径;非法转换拒绝
|
||||
16
.scratch/v1.0/issues/03-session-api.md
Normal file
16
.scratch/v1.0/issues/03-session-api.md
Normal file
@ -0,0 +1,16 @@
|
||||
# 03 — 智能评估会话生命周期 API
|
||||
|
||||
**What to build:** 会话的创建、消息收发(含通道转发)、关闭 API。OpenClaw 通过这组接口驱动虚拟用户与被评对象的对话。
|
||||
|
||||
**Blocked by:** 02(评估状态必须为 executing 才能创建会话)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `POST /api/intelligent-evals/{id}/sessions`:创建会话(body = persona, goal, dimension),仅 executing 状态可创建
|
||||
- [x] `POST /api/intelligent-evals/{id}/sessions/{sid}/messages`:发送消息(body = content),平台通过 ChannelFactory 转发到被评对象,持久化双方消息,返回回复内容与 latency_ms
|
||||
- [x] `POST /api/intelligent-evals/{id}/sessions/{sid}/close`:关闭会话(body = verdict JSON),会话转 completed,记录 closed_at
|
||||
- [x] `GET /api/intelligent-evals/{id}/sessions`:会话列表
|
||||
- [x] `GET /api/intelligent-evals/{id}/sessions/{sid}/messages`:单会话消息记录
|
||||
- [x] 会话非 running 状态发消息返回 409
|
||||
- [x] 评估非 executing 状态创建会话返回 409
|
||||
- [x] turn_count 随消息自动递增
|
||||
- [x] 集成测试覆盖:会话生命周期(创建 → 对话 → 关闭)、状态拒绝、消息持久化
|
||||
15
.scratch/v1.0/issues/04-report-api.md
Normal file
15
.scratch/v1.0/issues/04-report-api.md
Normal file
@ -0,0 +1,15 @@
|
||||
# 04 — 智能评估报告 API + Markdown 导出
|
||||
|
||||
**What to build:** OpenClaw 提交结构化报告的接口、报告读取接口、Markdown 渲染导出。
|
||||
|
||||
**Blocked by:** 03(会话完成才有报告)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `PUT /api/intelligent-evals/{id}/report`:OpenClaw 提交结构化报告(body = report JSON),状态 executing → completed,记录 completed_at
|
||||
- [x] `GET /api/intelligent-evals/{id}/report`:读取报告
|
||||
- [x] `GET /api/intelligent-evals/{id}/report/markdown`:Markdown 渲染导出
|
||||
- [x] 报告结构校验:必须包含 summary、findings 数组;findings 每项必须有 issue、severity、dimension
|
||||
- [x] 评估非 executing 状态提交报告返回 409
|
||||
- [x] Markdown 渲染为纯函数(dict 进 string 出),有单测
|
||||
- [x] 无报告时 GET report 返回空对象或 404
|
||||
- [x] 集成测试覆盖:提交报告 → 状态转 completed → 读取 → Markdown 导出
|
||||
13
.scratch/v1.0/issues/05-frontend-list.md
Normal file
13
.scratch/v1.0/issues/05-frontend-list.md
Normal file
@ -0,0 +1,13 @@
|
||||
# 05 — 前端:智能评估列表页 + 创建表单
|
||||
|
||||
**What to build:** 替换占位页,实现智能评估列表展示和创建表单。
|
||||
|
||||
**Blocked by:** 02(API 就绪)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] 列表页:Table 展示所有智能评估(名称、对象、状态、创建时间),状态用 Tag 着色
|
||||
- [x] 创建表单(Modal 或 Drawer):名称、评测对象选择、目标(一句话)、种子集(JSON 编辑或简化表单)、考察意图、角色描述、时间窗口(小时数)
|
||||
- [x] 创建成功后列表刷新
|
||||
- [x] 列表行可点击进入详情(路由跳转或内部状态切换)
|
||||
- [x] 空状态提示
|
||||
- [x] `tsc --noEmit` 通过
|
||||
14
.scratch/v1.0/issues/06-frontend-detail.md
Normal file
14
.scratch/v1.0/issues/06-frontend-detail.md
Normal file
@ -0,0 +1,14 @@
|
||||
# 06 — 前端:智能评估管理/详情页
|
||||
|
||||
**What to build:** 智能评估的管理详情页——展示状态、用户输入、粗计划(含审批操作)、执行进度。
|
||||
|
||||
**Blocked by:** 02, 03(API 就绪)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] 展示评估基本信息(名称、对象、状态、时间窗口)
|
||||
- [x] 展示用户输入四件套(目标、种子、意图、角色描述)
|
||||
- [x] pending_approval 状态:展示粗计划(维度、虚拟用户、时间分布、预算、完成标准),提供「批准」和「打回」按钮(打回需填反馈)
|
||||
- [x] executing 状态:展示执行进度(已完成会话数 / 预估会话数、时间分布进度)
|
||||
- [x] planning 状态:展示"规划中"等待提示
|
||||
- [x] 从列表页可进入此页面
|
||||
- [x] `tsc --noEmit` 通过
|
||||
15
.scratch/v1.0/issues/07-frontend-report.md
Normal file
15
.scratch/v1.0/issues/07-frontend-report.md
Normal file
@ -0,0 +1,15 @@
|
||||
# 07 — 前端:报告页 + 会话下钻
|
||||
|
||||
**What to build:** 结构化报告展示页面和会话对话下钻功能。
|
||||
|
||||
**Blocked by:** 04(报告 API 就绪)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] 报告页:总览摘要、发现清单(问题+严重程度+维度+建议)、亮点、优先级建议
|
||||
- [x] 发现清单按严重程度排序(high → medium → low)
|
||||
- [x] 每个发现可展开查看证据(对话轮次引用)
|
||||
- [x] 会话列表:展示所有会话(人设、目标、维度、状态、轮数)
|
||||
- [x] 会话下钻:点击会话查看完整对话记录(气泡样式)+ 会话级评估(verdict)
|
||||
- [x] Markdown 导出按钮(调 report/markdown API 下载)
|
||||
- [x] 从详情页/列表页可进入报告页
|
||||
- [x] `tsc --noEmit` 通过
|
||||
13
.scratch/v1.0/issues/08-openclaw-skills.md
Normal file
13
.scratch/v1.0/issues/08-openclaw-skills.md
Normal file
@ -0,0 +1,13 @@
|
||||
# 08 — OpenClaw 技能:planner + evaluator + analyst
|
||||
|
||||
**What to build:** 三个 OpenClaw 技能文件(按角色拆分),走部署脚本既有 skill 同步管线分发。
|
||||
|
||||
**Blocked by:** 02, 03, 04(全部 API 就绪才能写技能说明书)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `agenteval-intelligent-planner` skill:指导 OpenClaw 读取评估输入(goal/seeds/intent/role_description)→ 产出粗计划 JSON → 调 `PUT /api/intelligent-evals/{id}/plan` 提交
|
||||
- [x] `agenteval-intelligent-evaluator` skill:指导 OpenClaw 按时间分布自唤醒 → 调 `POST .../sessions` 创建会话 → 循环调 `POST .../messages` 对话 → 调 `POST .../close` 关闭并提交 verdict
|
||||
- [x] `agenteval-intelligent-analyst` skill:指导 OpenClaw 读取所有会话结果 → 汇总产出结构化报告 JSON → 调 `PUT /api/intelligent-evals/{id}/report` 提交
|
||||
- [x] 三个技能文件放在 `backend/plugins/openclaw/skills/` 下
|
||||
- [x] 部署脚本的 skill 同步步骤覆盖新技能
|
||||
- [x] t480 端到端验证:创建一个智能评估 → OpenClaw 产出计划 → 审批 → 执行 → 报告落库
|
||||
262
.scratch/v1.0/spec.md
Normal file
262
.scratch/v1.0/spec.md
Normal file
@ -0,0 +1,262 @@
|
||||
# v1.0 规格说明:智能评估第一期 —— OpenClaw 驱动的独立评测体系
|
||||
|
||||
**状态**: draft
|
||||
**日期**: 2026-08-04
|
||||
**决策依据**: grilling 共识清单(16 项)、ADR-0003 v2 修订、CONTEXT.md「探索式评测」章节
|
||||
|
||||
## Problem Statement
|
||||
|
||||
平台的评测能力(v0.8 静态评估)全部建立在"固定场景(考纲)"之上:预设题目、预设规则、统计通过率。这能回答"考纲过了多少",但无法模拟真实用户的复杂行为——用户不按考纲出牌,他们会改主意、追问、被绕晕、放弃。
|
||||
|
||||
v0.9 把探索式评测嵌入了现有活动作为增强层,但探索仍是"活动内的附加",没有独立生命周期。
|
||||
|
||||
本期引入**智能评估**——与静态评估并列的独立评测体系:用户只给方向(目标+种子+意图+角色),OpenClaw 全权规划与执行,产出可驱动被评对象改善的结构化报告。
|
||||
|
||||
## Solution
|
||||
|
||||
### 体系定位
|
||||
|
||||
两套评测体系正式分家:
|
||||
|
||||
| | 静态评估(v0.8) | 智能评估(v1.0) |
|
||||
|---|---|---|
|
||||
| 驱动者 | 平台调度器 | OpenClaw |
|
||||
| 输入 | 固定场景(考纲) | 目标+种子+意图+角色 |
|
||||
| 判定 | 规则驱动(关键词/时延/LLM打分) | OpenClaw 自主判定 |
|
||||
| 产出 | 通过率、对比报告 | 结构化发现报告(可驱动被评对象进化) |
|
||||
| 数据模型 | Campaign → Run → Turn → Result | IntelligentEval → Session → Message → Verdict |
|
||||
|
||||
### 核心概念
|
||||
|
||||
- **智能评估(IntelligentEval)**:独立实体,与 Campaign 平级。单对象绑定,拥有自己的状态机、配置、报告结构
|
||||
- **粗计划(Coarse Plan)**:OpenClaw 产出的评估规划——评估维度、虚拟用户、时间分布、预算。入库可见,用户审批后才执行
|
||||
- **智能评估会话(IntelligentEvalSession)**:OpenClaw 以虚拟用户身份与被评对象的一段完整对话,全新实体不复用 exploration_sessions
|
||||
- **结构化报告**:发现清单+证据+严重程度+改善建议,可喂给被评对象的提示词/SOP 改善流程
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As an 评测平台用户, I want 创建智能评估时提供目标、种子集、考察意图和 OpenClaw 角色描述, so that OpenClaw 有方向但不被锁死路径
|
||||
2. As an 评测平台用户, I want 看到 OpenClaw 产出的粗计划并决定是否批准, so that 评估方向在我掌控之中
|
||||
3. As an 评测平台用户, I want 打回不满意的计划并附反馈, so that OpenClaw 能根据反馈重新规划
|
||||
4. As an 评测平台用户, I want 在管理页面看到执行进度, so that 我知道评估走到哪了
|
||||
5. As an 评测平台用户, I want 评估完成后看到结构化报告(发现清单、证据、建议), so that 我知道被评对象的问题在哪
|
||||
6. As an 评测平台用户, I want 点开单个会话查看完整对话, so that 每个"发现"都能下钻到原文证据
|
||||
7. As an 评测平台用户, I want 取消正在执行的评估, so that 方向错误时能及时止损
|
||||
8. As an 评测平台用户, I want 导出报告为 Markdown, so that 离线分享材料完整
|
||||
9. As an OpenClaw planner, I want 收到评估输入后产出粗计划并通过 API 提交, so that 用户能审批我的规划
|
||||
10. As an OpenClaw evaluator, I want 按时间分布自唤醒执行会话, so that 交互合理分布在模拟的服务周期内
|
||||
11. As an OpenClaw analyst, I want 汇总所有会话结果产出结构化报告, so that 发现可驱动被评对象改善
|
||||
12. As an 平台, I want OpenClaw 停摆时智能评估标记 failed 而非无限等待, so that 不留悬挂评估
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
### 数据模型
|
||||
|
||||
**IntelligentEval(智能评估)**——独立实体,与 Campaign 平级:
|
||||
|
||||
```
|
||||
IntelligentEval {
|
||||
id: str (uuid)
|
||||
name: str // 评估名称
|
||||
target_id: str // 绑定的评测对象(单对象)
|
||||
status: IntelligentEvalStatus // 状态机
|
||||
|
||||
// 用户输入(四件套)
|
||||
goal: str // 一句话目标
|
||||
seeds: JSON // 种子集(人设数组 + 目标数组)
|
||||
intent: str // 考察意图
|
||||
role_description: str // OpenClaw 角色描述
|
||||
|
||||
// 粗计划(OpenClaw 产出)
|
||||
plan: JSON | null // 评估维度、虚拟用户、时间分布、预算、完成标准
|
||||
plan_feedback: str | null // 打回时用户的反馈
|
||||
|
||||
// 时间窗口
|
||||
time_window_hours: int // 模拟服务周期(如 24)
|
||||
|
||||
// 报告
|
||||
report: JSON | null // 结构化报告(发现清单+证据+建议)
|
||||
|
||||
// 时间戳
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
started_at: datetime | null // 批准执行时
|
||||
completed_at: datetime | null // 完成/取消/失败时
|
||||
}
|
||||
```
|
||||
|
||||
**IntelligentEvalSession(智能评估会话)**——全新实体:
|
||||
|
||||
```
|
||||
IntelligentEvalSession {
|
||||
id: str (uuid)
|
||||
eval_id: str // 归属的智能评估
|
||||
target_id: str // 被评对象
|
||||
persona: JSON // 虚拟用户人设
|
||||
goal: str // 会话目标
|
||||
dimension: str | null // 所属评估维度
|
||||
status: SessionStatus // running / completed / failed / expired
|
||||
|
||||
// 会话级评估(OpenClaw 关闭时提交)
|
||||
verdict: JSON | null // 结构化评估(达成度、问题、建议)
|
||||
|
||||
turn_count: int
|
||||
created_at: datetime
|
||||
closed_at: datetime | null
|
||||
}
|
||||
```
|
||||
|
||||
**IntelligentEvalMessage(会话消息)**:
|
||||
|
||||
```
|
||||
IntelligentEvalMessage {
|
||||
id: str (uuid)
|
||||
session_id: str // 归属会话
|
||||
role: str // "user" | "assistant"
|
||||
content: str // 消息内容
|
||||
latency_ms: int | null // 被评对象回复延迟
|
||||
created_at: datetime
|
||||
}
|
||||
```
|
||||
|
||||
**状态机**:
|
||||
|
||||
```
|
||||
IntelligentEvalStatus:
|
||||
draft → planning → pending_approval → executing → completed
|
||||
→ cancelled
|
||||
→ failed
|
||||
|
||||
pending_approval 可打回 → planning(附反馈)
|
||||
executing 可取消 → cancelled
|
||||
```
|
||||
|
||||
### API 契约(全部 X-API-Key 鉴权,沿用 `require_api_key`)
|
||||
|
||||
**规划阶段:**
|
||||
- `POST /api/intelligent-evals`:创建智能评估(body = name, target_id, goal, seeds, intent, role_description, time_window_hours)→ 状态 draft → planning
|
||||
- `PUT /api/intelligent-evals/{id}/plan`:OpenClaw 提交粗计划(body = plan JSON)→ 状态 planning → pending_approval
|
||||
|
||||
**审批阶段:**
|
||||
- `POST /api/intelligent-evals/{id}/approve`:用户批准 → 状态 pending_approval → executing,记录 started_at
|
||||
- `POST /api/intelligent-evals/{id}/reject`:用户打回(body = feedback)→ 状态 pending_approval → planning
|
||||
|
||||
**执行阶段(OpenClaw 调用):**
|
||||
- `GET /api/intelligent-evals/{id}`:读取评估配置与计划
|
||||
- `POST /api/intelligent-evals/{id}/sessions`:创建会话(body = persona, goal, dimension)→ 返回 session_id
|
||||
- `POST /api/intelligent-evals/{id}/sessions/{sid}/messages`:发消息(body = content)→ 平台转发到被评对象通道,返回回复与延迟
|
||||
- `POST /api/intelligent-evals/{id}/sessions/{sid}/close`:关闭会话(body = verdict JSON)→ 状态转 completed
|
||||
|
||||
**收尾阶段:**
|
||||
- `PUT /api/intelligent-evals/{id}/report`:OpenClaw 提交结构化报告 → 状态 executing → completed
|
||||
- `POST /api/intelligent-evals/{id}/cancel`:用户取消 → 状态 → cancelled
|
||||
|
||||
**读出口(前端用):**
|
||||
- `GET /api/intelligent-evals`:列表
|
||||
- `GET /api/intelligent-evals/{id}`:详情(含计划、进度)
|
||||
- `GET /api/intelligent-evals/{id}/sessions`:会话列表
|
||||
- `GET /api/intelligent-evals/{id}/sessions/{sid}/messages`:单会话对话记录
|
||||
- `GET /api/intelligent-evals/{id}/report`:结构化报告
|
||||
- `GET /api/intelligent-evals/{id}/report/markdown`:Markdown 导出
|
||||
|
||||
### 通道复用
|
||||
|
||||
智能评估的消息转发复用现有 ChannelFactory——通过 target_id 找到对应的通道配置,用同一套 send/poll 机制与被评对象交互。不新建通道。
|
||||
|
||||
### 护栏
|
||||
|
||||
第一版信任 OpenClaw 自律:粗计划里约定的预算(会话数、轮数)就是契约,平台不做硬校验。后续版本可加硬护栏(超了返 409)。
|
||||
|
||||
### OpenClaw 技能(按角色拆分)
|
||||
|
||||
三个独立技能文件,走部署脚本既有 skill 同步管线分发:
|
||||
|
||||
| 技能 | 角色 | 职责 | 触发方式 |
|
||||
|------|------|------|---------|
|
||||
| `agenteval-intelligent-planner` | 规划师 | 收到评估输入 → 产出粗计划 → 调 API 提交 | 平台状态转 planning 时唤醒 |
|
||||
| `agenteval-intelligent-evaluator` | 评估者 | 按时间分布自唤醒 → 创建会话 → 对话 → 关闭 → 提交会话评估 | OpenClaw cron 自唤醒 |
|
||||
| `agenteval-intelligent-analyst` | 分析师 | 所有会话完成 → 汇总产出结构化报告 → 调 API 提交 | 平台状态检测到所有会话完成时唤醒 |
|
||||
|
||||
每个角色可配置不同的大模型(规划师用推理强的、评估者用对话自然的、分析师用结构化输出好的)。
|
||||
|
||||
### 粗计划结构
|
||||
|
||||
```json
|
||||
{
|
||||
"dimensions": ["退货流程", "投诉处理", "多轮追问"],
|
||||
"virtual_users": [
|
||||
{ "persona": { "background": "...", "personality": "...", "patience": "low" }, "goal": "完成退货" },
|
||||
{ "persona": { "background": "...", "personality": "...", "patience": "high" }, "goal": "了解政策" }
|
||||
],
|
||||
"time_distribution": [
|
||||
{ "time_slot": "0-2h", "sessions": 1, "scenario": "早间咨询" },
|
||||
{ "time_slot": "8-10h", "sessions": 2, "scenario": "工作时段高峰" },
|
||||
{ "time_slot": "18-22h", "sessions": 2, "scenario": "晚间投诉" }
|
||||
],
|
||||
"estimated_sessions": 5,
|
||||
"budget": { "max_turns_per_session": 12, "total_max_turns": 60 },
|
||||
"completion_criteria": "每个维度至少一个会话产出评估"
|
||||
}
|
||||
```
|
||||
|
||||
### 结构化报告模板(第一版)
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "一段话总结整体表现",
|
||||
"scores": { "维度": 分数 } | null,
|
||||
"findings": [
|
||||
{
|
||||
"issue": "退货流程中未主动确认订单号",
|
||||
"severity": "high" | "medium" | "low",
|
||||
"dimension": "退货流程",
|
||||
"evidence": [
|
||||
{ "session_id": "...", "turn_index": 3, "user_said": "...", "assistant_replied": "..." }
|
||||
],
|
||||
"suggestion": "在退货意图识别后,增加订单号确认步骤",
|
||||
"related_sop": "退货处理流程 §3.2" | null
|
||||
}
|
||||
],
|
||||
"highlights": [
|
||||
{ "description": "多轮追问中保持了上下文连贯性", "dimension": "多轮追问" }
|
||||
],
|
||||
"priority_recommendations": ["先修 退货确认 问题,再优化 投诉共情"]
|
||||
}
|
||||
```
|
||||
|
||||
### 前端
|
||||
|
||||
独立顶级页面「智能评估」,四个子页面:
|
||||
|
||||
1. **列表页**:所有智能评估的列表(名称、对象、状态、创建时间),可新建
|
||||
2. **管理页**:展示当前状态、用户输入、粗计划(待审批时可批准/打回)、执行进度
|
||||
3. **报告页**:结构化报告展示——发现清单(问题+证据+建议)、亮点、会话列表
|
||||
4. **会话详情页**:单个会话的完整对话记录 + 会话级评估
|
||||
|
||||
### 时间窗口语义
|
||||
|
||||
时间窗口是**模拟约束**而非硬截止。"模拟 24 小时服务周期内的用户交互"——OpenClaw 规划时要考虑交互的时间分布(早高峰、午间冷清、晚间投诉多),合理分布会话时机。OpenClaw 通过自身 cron 机制在对应时间点自唤醒执行。
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- 好测试只测外部行为:API 契约、状态机迁移、报告渲染;不测 OpenClaw 内部决策
|
||||
- **集成测试(TestClient,主力接缝)**:智能评估生命周期(创建→规划→审批→执行→报告)、状态机各分支(打回、取消)、会话生命周期(创建→对话→关闭)
|
||||
- **纯函数单测**:Markdown 渲染(dict 进 string 出)
|
||||
- 前端仅 `tsc --noEmit`(沿用现状)
|
||||
- OpenClaw 技能行为不进自动化测试,靠部署后端到端验证(与 v0.4 以来 skill 验收方式一致)
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 多对象绑定(第一版单对象,后续按需扩展)
|
||||
- 平台硬护栏(第一版信任 OpenClaw 自律,后续可加 409 拒绝)
|
||||
- 跨智能评估对比(多期智能评估的横向对照)
|
||||
- 报告模板可配置(第一版固定模板)
|
||||
- 事件驱动唤醒(纯 cron 自唤醒)
|
||||
- 菜单二级化重构(稍后单独处理)
|
||||
|
||||
## Further Notes
|
||||
|
||||
- 智能评估与静态评估完全独立:不共享数据表、不共享状态机、不共享报告结构
|
||||
- OpenClaw 三个角色技能可配不同大模型,模型配置走现有 model_configs 体系
|
||||
- 时间分布编排由 OpenClaw 在粗计划中决定,平台只存储不校验
|
||||
- 报告的消费者有两个:人(看问题)和 AI(拿报告去改提示词/SOP)——结构化是核心要求
|
||||
@ -74,6 +74,8 @@ const routeConfigs: RouteConfig[] = [
|
||||
const componentMap: Record<string, () => ReactNode> = {}
|
||||
routeConfigs.forEach((r) => { componentMap[r.path] = r.component })
|
||||
|
||||
const PINNED_TAB_PATHS = new Set(['/', '/openclaw'])
|
||||
|
||||
const menuItems: MenuProps['items'] = [
|
||||
{ key: '/', icon: <DashboardOutlined />, label: '仪表盘' },
|
||||
{ key: '/openclaw', icon: <RobotOutlined />, label: 'AI 助手' },
|
||||
@ -147,7 +149,7 @@ function App() {
|
||||
key: config.path,
|
||||
title: config.name,
|
||||
icon: config.icon,
|
||||
closable: config.path !== '/',
|
||||
closable: !PINNED_TAB_PATHS.has(config.path),
|
||||
}
|
||||
openTab(tab)
|
||||
} else {
|
||||
@ -163,7 +165,7 @@ function App() {
|
||||
key: config.path,
|
||||
title: config.name,
|
||||
icon: config.icon,
|
||||
closable: config.path !== '/',
|
||||
closable: !PINNED_TAB_PATHS.has(config.path),
|
||||
}
|
||||
openTab(tab)
|
||||
navigate(key)
|
||||
|
||||
@ -1,12 +1,12 @@
|
||||
import { useState } from 'react'
|
||||
import { useEffect, useState } from 'react'
|
||||
import {
|
||||
Alert, Button, Card, Descriptions, Empty, Input, Modal, Popconfirm, Space, Spin, Tag, message,
|
||||
} from 'antd'
|
||||
import { ArrowLeftOutlined, FileTextOutlined, StopOutlined } from '@ant-design/icons'
|
||||
import { intelligentEvalsApi, type IntelligentEval } from '../../api'
|
||||
import { intelligentEvalsApi, type IntelligentEval, type IntelligentEvalSession } from '../../api'
|
||||
import { colors } from '../../tokens'
|
||||
import { formatDateTime, shortDateTime } from '../../utils/date'
|
||||
import { EVAL_STATUS } from './status'
|
||||
import { EVAL_STATUS, SESSION_STATUS } from './status'
|
||||
|
||||
const sectionCard: React.CSSProperties = { marginBottom: 16 }
|
||||
|
||||
@ -71,7 +71,21 @@ export default function EvalDetail({ ev, targetName, onBack, onOpenReport, onCha
|
||||
const [busy, setBusy] = useState(false)
|
||||
const [rejectOpen, setRejectOpen] = useState(false)
|
||||
const [feedback, setFeedback] = useState('')
|
||||
const [sessions, setSessions] = useState<IntelligentEvalSession[] | null>(null)
|
||||
const meta = EVAL_STATUS[ev.status] ?? { label: ev.status, color: 'default' }
|
||||
const showSessions = ev.status === 'executing' || ev.status === 'completed'
|
||||
|
||||
useEffect(() => {
|
||||
if (!showSessions) { setSessions(null); return }
|
||||
let cancelled = false
|
||||
const load = () => intelligentEvalsApi.listSessions(ev.id)
|
||||
.then((res) => { if (!cancelled) setSessions(res.data.sessions) })
|
||||
.catch(() => undefined)
|
||||
void load()
|
||||
if (ev.status !== 'executing') return () => { cancelled = true }
|
||||
const timer = setInterval(load, 5000)
|
||||
return () => { cancelled = true; clearInterval(timer) }
|
||||
}, [ev.id, ev.status, showSessions])
|
||||
|
||||
const runAction = async (fn: () => Promise<unknown>, okMsg: string) => {
|
||||
setBusy(true)
|
||||
@ -177,15 +191,40 @@ export default function EvalDetail({ ev, targetName, onBack, onOpenReport, onCha
|
||||
</Card>
|
||||
)}
|
||||
|
||||
{showSessions && (
|
||||
<Card size="small" title={`会话进度(${ev.completed_sessions}/${ev.session_count})`} style={sectionCard}>
|
||||
{sessions === null && <Spin size="small" />}
|
||||
{sessions !== null && sessions.length === 0 && (
|
||||
<div style={{ fontSize: 13, color: colors.textSecondary }}>
|
||||
等待 OpenClaw 按粗计划的时间分布创建会话…
|
||||
</div>
|
||||
)}
|
||||
{sessions?.map((s) => {
|
||||
const sMeta = SESSION_STATUS[s.status]
|
||||
return (
|
||||
<div
|
||||
key={s.id}
|
||||
style={{
|
||||
display: 'flex', alignItems: 'center', gap: 8, flexWrap: 'wrap',
|
||||
padding: '6px 0', borderBottom: `1px solid ${colors.border}`, fontSize: 13,
|
||||
}}
|
||||
>
|
||||
<b style={{ color: colors.text }}>{personaLabel(s.persona)}</b>
|
||||
<Tag color={sMeta.color} style={{ margin: 0 }}>{sMeta.label}</Tag>
|
||||
{s.dimension && <Tag style={{ margin: 0 }}>{s.dimension}</Tag>}
|
||||
<span style={{ color: colors.textSecondary, flex: 1, minWidth: 120 }}>{s.goal}</span>
|
||||
<span style={{ color: colors.textMuted }}>{s.turn_count} 轮</span>
|
||||
{s.created_at && (
|
||||
<span style={{ color: colors.textMuted }}>{shortDateTime(s.created_at)}</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
{ev.status === 'executing' && (
|
||||
<Card size="small" title="执行进度" style={sectionCard}>
|
||||
<div style={{ fontSize: 13, color: colors.textSecondary, marginBottom: 8 }}>
|
||||
已完成会话 {ev.completed_sessions} / 已创建 {ev.session_count}
|
||||
{ev.plan?.estimated_sessions ? ` / 预估 ${ev.plan.estimated_sessions}` : ''}
|
||||
</div>
|
||||
<div style={{ fontSize: 12, color: colors.textMuted }}>
|
||||
会话由 OpenClaw 按粗计划的时间分布自唤醒创建,最近活动于 {shortDateTime(ev.updated_at)}
|
||||
<div style={{ fontSize: 12, color: colors.textMuted, marginTop: 8 }}>
|
||||
会话由 OpenClaw 按粗计划的时间分布自唤醒创建,本页每 5 秒自动刷新
|
||||
</div>
|
||||
)}
|
||||
</Card>
|
||||
)}
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
import { useState } from 'react'
|
||||
import { useEffect, useState } from 'react'
|
||||
import {
|
||||
Button, Drawer, Empty, Form, Input, InputNumber, Select, Space, Table, Tag, message,
|
||||
} from 'antd'
|
||||
@ -45,11 +45,20 @@ export default function IntelligentEvalsPage() {
|
||||
{ tabPath: '/intelligent-evals' },
|
||||
)
|
||||
|
||||
const { data: selected } = useResource(
|
||||
const { data: selected, reload: reloadSelected } = useResource(
|
||||
() => (selectedId ? intelligentEvalsApi.get(selectedId).then((r) => r.data) : Promise.resolve(null)),
|
||||
{ deps: [selectedId, detailTick] },
|
||||
)
|
||||
|
||||
const pollActive = view === 'detail' && selected != null
|
||||
&& (selected.status === 'planning' || selected.status === 'executing' || selected.status === 'pending_approval')
|
||||
|
||||
useEffect(() => {
|
||||
if (!pollActive) return
|
||||
const timer = setInterval(() => void reloadSelected(true), 5000)
|
||||
return () => clearInterval(timer)
|
||||
}, [pollActive, reloadSelected])
|
||||
|
||||
const targetName = (id: string) =>
|
||||
targets?.find((t) => t.id === id)?.name ?? id.slice(0, 8)
|
||||
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
import { create } from 'zustand'
|
||||
import { RobotOutlined } from '@ant-design/icons'
|
||||
|
||||
export interface TabItem {
|
||||
key: string
|
||||
@ -16,7 +17,10 @@ interface TabStore {
|
||||
}
|
||||
|
||||
export const useTabStore = create<TabStore>((set, get) => ({
|
||||
tabs: [{ key: '/', title: '仪表盘', closable: false }],
|
||||
tabs: [
|
||||
{ key: '/', title: '仪表盘', closable: false },
|
||||
{ key: '/openclaw', title: 'AI 助手', icon: <RobotOutlined />, closable: false },
|
||||
],
|
||||
activeKey: '/',
|
||||
|
||||
openTab: (tab) => {
|
||||
@ -18,10 +18,10 @@ from agenteval.models import (
|
||||
Campaign,
|
||||
CampaignPlanEntry,
|
||||
CampaignStatus,
|
||||
EvalTarget,
|
||||
Scenario,
|
||||
Case,
|
||||
CaseType,
|
||||
EvalTarget,
|
||||
Scenario,
|
||||
)
|
||||
from agenteval.storage.repository import (
|
||||
CampaignAnalysisRepository,
|
||||
|
||||
Loading…
Reference in New Issue
Block a user