solahqb/AgentEvalTool
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
176 Commits 13 Branches 4 Tags 6 MiB
37da87c3b0
Commit Graph

25 Commits

Author SHA1 Message Date
sinohqb
37da87c3b0 refactor(intelligent-eval): openclaw_client token from settings (S5) + ADR-0008
Some checks failed
CI / test (push) Failing after 4m30s
Details 
P4 boundary (issue #10):
- S5: openclaw_client reads gateway_token and container_name from
  Settings (AGENTEVAL_OPENCLAW_GATEWAY_TOKEN / CONTAINER_NAME), with
  backwards-compatible defaults. Test injection still works via __init__.
- ADR-0008 documents the full deepening: S1 (domain convergence), S4
  (stuck-task settlement), S2 (router → service), S3 deferral rationale,
  S5 (token config), S6 (frontend polling deferred).

S6 (CronPoolMonitor unified polling) deferred to independent issue.
No behaviour change — 873 passed + 5 xfailed unchanged.
 2026-08-13 14:20:49 +08:00
sinohqb
617cc3909d chore: clean up release notes and update lockfile 
- Remove old release-notes-v1.1.md (renamed to v1.1-legacy.md)
- Update package-lock.json after version bump to 1.1.0
 2026-08-12 11:12:29 +08:00
sinohqb
6f2be0e68f docs(release): add v1.1.0 release notes 
Document the intelligent evaluation cron pool architecture:
- Cron pool management (5-20 workers)
- Task queue with priority scheduling
- Worker skill with autonomous decision logic
- Config snapshot management and comparison
- Decision process tracking and visualization
- Monitoring and alerting system
- Fault tolerance and recovery mechanisms
- Frontend UI for config history, decision process, and cron pool monitoring

Rename legacy v1.1 release notes to v1.1-legacy.md.
 2026-08-12 11:11:34 +08:00
sinohqb
1aa453ef0a feat(intelligent-eval): add cron pool data model and task queue API 
Implement Ticket 01 of intelligent eval cron pool architecture (ADR-0007):

- Add 4 new tables: task_queue, cron_pool, config_snapshots, decision_logs
- Implement task enqueueing logic with priority calculation
- Implement task assignment and completion APIs
- Add unit tests (9) and integration tests (7)
- Update CONTEXT.md with new vocabulary
- Add ADR-0007 documenting cron pool architecture decision

All 760 tests passing.
 2026-08-12 02:13:21 +08:00
sinohqb
941df4c8c7 release: normalize project version to v1.0.0 2026-08-11 14:07:05 +08:00
sinohqb
1782b245bf refactor(architecture): deepen campaign runtime modules 2026-08-11 13:18:48 +08:00
sinohqb
10a089e740 docs(deploy): record production access topology
Some checks failed
CI / test (push) Failing after 1m31s
Details
2026-08-09 03:49:06 +08:00
sinohqb
855455c0b4 feat(deploy): add production release workflow 2026-08-07 15:53:36 +08:00
sinohqb
62485684ca fix(architecture): enforce lifecycle consistency 2026-08-07 10:59:27 +08:00
sinohqb
c896ab3f71 refactor(architecture): deepen evaluation lifecycle and read model 2026-08-07 03:11:37 +08:00
sinohqb
f26c34a340 docs(ui-consistency): archive completed refactor, record drawer evolution in ADR-0005
Some checks failed
CI / test (push) Failing after 34s
Details
2026-08-06 02:01:36 +08:00
sinohqb
d4c254e121 refactor(ui): baseline FormDrawer component and IntelligentEvals dogfood (ui-consistency ticket 01) 
Extract the standard form drawer (width 640, footer actions, destroyOnClose)
as FormDrawer; reuse existing usePolling hook. IntelligentEvals becomes the
first consumer — create drawer and both 5s pollings switched with zero
behaviour change. Includes ui-consistency spec, tickets, and ADR-0005.
 2026-08-05 15:34:31 +08:00
sinohqb
c35159ffcb docs(v2): record exploratory-evaluation consensus in CONTEXT.md and ADR-0003 
Grilling 沉淀：OpenClaw 定位从"重规划"转为"虚拟用户"双轨结构
（周期活动内嵌探索先行 + 探索活动后续里程碑），新增六个领域词条。
 2026-08-03 16:40:28 +08:00
sinohqb
2d7679a6bb docs(release): freeze v0.8「比」period comparison milestone as 0.8.0
Some checks failed
CI / test (push) Failing after 49s
Details 
周期对比发布说明：计划指纹自动基线、机械 diff + LLM 演进叙述、
抽屉对比区块、正式线自动链，及发布后评审修复、导出纳入与 ruff 清理。
 2026-08-03 15:24:07 +08:00
sinohqb
fca3d64270 docs(release): freeze v0.6「巡」/ v0.7「析」milestone as 0.7.0 
Record the Campaign milestone (v0.6: durable scheduler, dual-axis
report, campaigns page) and the campaign-intelligence milestone (v0.7:
two-phase analysis agent, analysis model default/override, drawer
section, auto-trigger and markdown export). Bump the single-source
version to 0.7.0.
 2026-08-03 02:42:47 +08:00
sinohqb
782916a283 refactor(metrics): type Run summary and converge cross-run aggregation 
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
 2026-07-31 01:57:56 +08:00
sinohqb
fe2a3f7579 docs(domain): add Campaign glossary terms and phasing ADR
Some checks failed
CI / test (push) Failing after 40s
Details 
领域建模拷问产出：CONTEXT.md 新增「周期评估」章节（评估活动 / 服务周期
窗口 / 活动计划）；ADR-0003 记录活动分期（v1 静态地基、v2 自适应回路）
与「OpenClaw 进调度热回路」的取舍。
 2026-07-30 11:12:57 +08:00
sinohqb
25b4c98dc8 docs(release): v0.5「准」发布说明与里程碑收尾
Some checks failed
CI / test (push) Has been cancelled
Details 
版本升至 0.5.0-dev；新增 release-notes-v0.5.md（判定语义 + 场景版本化
+ 领域文档基线）；AGENT.md/AGENTS.md 更新里程碑路线图、判定语义速查与
CONTEXT.md/ADR 指引。
 2026-07-29 15:43:15 +08:00
sinohqb
770d260750 feat(report): compare requires same scenario version (ticket 05)
Some checks failed
CI / test (push) Failing after 39s
Details 
对比报告可比性收紧为同场景同考纲版本（ADR-0001）：跨版本 API 返回 400
（detail 含双方版本号），报告生成层抛 ValueError；前端对比候选按
同场景 + 同版本过滤，A 变更后自动清空不可比的 B。文档"尚未实现"标注移除。
 2026-07-29 11:21:52 +08:00
sinohqb
e33922c3fe docs(domain): add domain glossary CONTEXT.md and first ADRs
Some checks failed
CI / test (push) Failing after 51s
Details 
拷问会话产出：14 条核心术语定义（评测对象/场景/用例/轮次/通过率/模型能力·用途等），
以及两项决策记录——场景版本化的可比性语义（ADR-0001）、通过率含执行失败的口径（ADR-0002）。
 2026-07-28 21:46:56 +08:00
sinohqb
595409487b docs(release): v0.4「联」发布说明与里程碑收尾
Some checks failed
CI / test (push) Failing after 38s
Details 
- 新增 release-notes-v0.4.md（事故排查 + 功能总结 + v0.5 候选方向）
- AGENT.md 里程碑表更新至 v0.4 + 鉴权配置说明
- README / plan-v0.4 状态同步
 2026-07-28 17:41:20 +08:00
sinohqb
92f98c3af7 docs(release): add v0.3 release notes and minimal CI baseline
Some checks failed
CI / test (push) Failing after 2m59s
Details 
- 补齐 release-notes-v0.3.md，更新 AGENT.md 里程碑表与 README 至 v0.3 现状
- 新增 scripts/ci-check.sh（版本一致性 + ruff + pytest + tsc）
- 新增 .gitea/workflows/ci.yml 与本地 pre-push hook 约定
 2026-07-27 16:51:45 +08:00
sinohqb
470ff5875f feat(models): add centralized model configuration 2026-07-17 20:02:43 +08:00
sinohqb
12481cd1b8 v0.3-s1: 规则层异步化 + 工具函数去重 + HTTP 通道 
## 核心变更

### 规则层全面异步化（DEBT-1）
- EvalRule.evaluate() 签名改为 async def，全量同步改造（无兼容层）
- LlmScoreRule._call_llm: requests.post → httpx.AsyncClient，彻底消除事件循环阻塞
- engine._save_rule_results: rule.evaluate() → await rule.evaluate()

### 工具函数去重（DEBT-2）
- 新建 agenteval/utils/llm.py，统一三个函数：
  - extract_reply_text (原 5 处重复)
  - extract_content_from_llm_response (原 2 处重复)
  - parse_json_from_llm_text (统一 LLM 输出 JSON 解析)
- engine.py / llm_score.py / runs.py / report.py 全部切换到 utils.llm

### HTTP 通用通道（S1-3）
- 新建 channels/http.py (HttpChannel)
  - 配置化 send_url / reply_url 模板 ({message}, {msg_id} 占位)
  - dot-path 提取 msg_id 和 reply_text
  - 可选 reply_ready_path 就绪标志
  - 长连接 AsyncClient 复用
- ChannelFactory 注册 ChannelType.HTTP → HttpChannel

### 测试
- 新增 tests/unit/test_http_channel_and_rules.py (19 个测试)
- _get_path / health_check / send / poll_reply / 超时 / 就绪标志 / async 规则评估
- 测试总数：24 → 43，全部通过

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-17 10:52:32 +08:00
sinohqb
a77cd83e6a v0.2.0-dev: 文件管理 + 页面布局统一 + 6 个 bug 修复 
## 新增功能
- 文件管理模块：分类树 + 文件上传/下载/删除
- 文件上传支持拖拽（Dragger）+ 手动上传（customRequest 模式）

## 页面布局统一（参照评测执行页）
- 仪表盘/评测对象/评测场景/评测报告 全部改为全高 flex 布局
- 统一内联页头样式（h2 + 竖线分隔 + 描述）
- 表格撑满高度、overflow 处理
- 每页添加刷新按钮

## Bug 修复
- 分类树操作按钮 hover 不可见（CSS 规则缺失）
- 文件上传失败（multipart boundary 缺失）
- LLM API 响应 content blocks 数组格式支持（_extract_content_from_api_response）
- response_time_max_ms 被静默忽略（隐式规则传空 params）
- 空 messages 导致 IndexError 崩溃
- poll_reply 异常中止整个 run（缺 try/catch）
- engine finally 未关闭 session
- 3 个页面 UTC 时间戳解析偏差 8 小时

## 后端
- EvalEngine: poll_reply 异常保护、空 dialog 保护、session 关闭
- LLM API 响应解析支持 content-block-array 格式
- 隐式 response_time 规则正确传递 max_ms 参数

## 前端
- api.ts: 移除手动 Content-Type（让浏览器自动添加 boundary）
- Files.tsx: customRequest 替代 beforeUpload、布局优化
- index.css: 分类树 hover 规则
- Targets/Scenarios/Home/Reports: 全高布局改造
- 3 个页面时间戳改用 formatDateTime()（修复 UTC 偏差）

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-16 15:25:22 +08:00