Commit Graph

21 Commits

Author SHA1 Message Date
sinohqb
b0969ae582 feat(intelligent-eval): backfill decision logs for completed evals
All checks were successful
CI / test (push) Successful in 4m2s
COMPLETED 状态的评估(历史/异常路径)可能完全没有决策日志,
导致旧报告决策过程为空。扩展 _supplement_decision_logs 支持
COMPLETED:按时段补 execute_session + 补 start_analysis(历史回填),
scan loop 每分钟自动回填,无需一次性脚本。幂等,只补缺失类型。
2026-08-17 13:06:27 +08:00
sinohqb
32f63e80ae style: ruff format (decision-log backfill)
Some checks failed
CI / test (push) Failing after 36s
2026-08-17 05:12:09 +08:00
sinohqb
25920280f6 feat(intelligent-eval): platform audit backfill for decision logs + disable legacy cron workers
Some checks failed
CI / test (push) Failing after 35s
- _supplement_decision_logs: executing evals missing a decision log get a
  platform-derived execute_session (deficit) or start_analysis (all sessions
  done) entry. Audit backfill only — records observable state, does not change
  agent execution. Called each scan tick after requeue+scan.
- t480 legacy cron workers (5) disabled: superseded by platform-triggered
  headless agent (plan C); they kept firing every minute and failing on
  Channel-required.
2026-08-17 05:11:18 +08:00
sinohqb
71c7cd3d39 fix(intelligent-eval): requeue stale assigned tasks (worker crash recovery)
All checks were successful
CI / test (push) Successful in 3m45s
方案③ worker 由平台触发 openclaw agent(cron=manual-run,非真实 cron),
fault_tolerance 的 stuck 检测不适用——agent 中断/失败时任务永久卡 assigned,
scan 只查 pending 不再入队(死锁)。

requeue_stale_assigned_tasks:assigned 超过 10 分钟且评估仍 executing 的
任务重置为 pending(清空认领),平台 scan 循环随后重新触发 worker 重试。
接入 scan loop,每轮先 requeue 再 scan。
2026-08-17 03:30:42 +08:00
sinohqb
8e65e2e7b0 fix(intelligent-eval): worker trigger message must demand immediate execution
All checks were successful
CI / test (push) Successful in 3m47s
openclaw agent has no cron state; a bare 'agenteval-intelligent-worker'
message made the worker skill decide then 'wait for the next tick',
deadlocking (task assigned, session never created). The trigger message now
demands '立即完成当前任务,不要等待下一节拍' and, when all sessions are
done, delegates to agenteval-intelligent-analyst. Verified end-to-end on
t480: 1h-window eval went executing -> session (2 real turns) -> close ->
report -> completed, fully agent-driven, no external IM channel.
2026-08-17 02:57:08 +08:00
sinohqb
775b070bab feat(intelligent-eval): platform triggers OpenClaw agent as worker (avoid external IM channel)
All checks were successful
CI / test (push) Successful in 3m48s
OpenClaw cron requires a channel (announce->last fail-closed); webchat is a
Control-UI feature, not an addressable channel, and platform-side static
execution would degrade the intelligent eval into a static evaluation.

Solution (plan C): the platform keeps the scan loop and, when the queue has
a pending task, invokes the headless agent:
  docker exec openclaw-eval openclaw agent --agent main \
    -m agenteval-intelligent-worker --json
--deliver defaults to false, so no cron delivery channel is involved. The
worker skill runs unchanged under the OpenClaw agent (LLM decisions +
evaluator/analyst skills). Verified headless invocation returns ok.
2026-08-17 02:37:37 +08:00
sinohqb
5836b84681 fix(intelligent-eval): add lifespan scan-loop for worker task enqueue
All checks were successful
CI / test (push) Successful in 3m53s
scan_and_enqueue_tasks had no scheduler: the OpenClaw Worker wakes every
minute but the platform never enqueued executing evals, so the queue was
always empty. lifespan now starts an asyncio background task that scans
executing intelligent evals every 60s (aligned with the Worker wake),
cancelled cleanly on shutdown. Verified by a new startup test (879 total).
2026-08-17 02:08:40 +08:00
sinohqb
2ff023a65b feat(intelligent-eval): implement cron pool management (ticket 02)
- Add OpenClawClient wrapping CLI commands (create/delete/list crons)
- Implement pool initialization, scale up/down, auto-scaling logic
- Implement cron state sync and stuck cron detection
- Add pool status and manual scaling APIs
- Add 13 unit tests and 5 integration tests

Pool automatically scales between 5-20 crons based on load.
All 778 tests passing.
2026-08-12 09:47:04 +08:00
sinohqb
1782b245bf refactor(architecture): deepen campaign runtime modules 2026-08-11 13:18:48 +08:00
sinohqb
c896ab3f71 refactor(architecture): deepen evaluation lifecycle and read model 2026-08-07 03:11:37 +08:00
sinohqb
1317552701 feat(intelligent-eval): add backend for OpenClaw-driven intelligent evaluation (tickets 01-04)
Introduce 智能评估 as an evaluation paradigm parallel to static evaluation,
driven by OpenClaw. The platform supplies storage, lifecycle, and reporting;
OpenClaw plans and executes.

- Data model: IntelligentEval + Session + Message tables (new, not reusing exploration)
- Lifecycle state machine: draft → planning → pending_approval → executing → completed/cancelled/failed
- Session API: create/message (channel-forwarded)/close with turn accounting
- Report API: pydantic-validated structured report, executing → completed, Markdown export (pure renderer)
- Alembic migration for the three tables; domain glossary added to CONTEXT.md
2026-08-05 03:18:52 +08:00
sinohqb
f3a528611e refactor(tasks): route LLM background tasks through TaskRegistry
架构保养第二轮候选 1:分析 / 周期对比 / judge 复核三条 LLM 任务链
收进各自的模块级 TaskRegistry(强引用防 GC、按 id 幂等、shutdown
统一收敛),删除 judge 的 _BACKGROUND_TASKS 私货,start_* 不再返回
无人消费的 Task。启动清理块补两笔 orphan 清扫:滞留的 generating
分析与对比行标记为 failed,与僵尸运行清扫同构。新增 7 个单测。
2026-08-04 09:58:20 +08:00
sinohqb
a351f65550 feat(exploration): session lifecycle endpoints with platform hard guardrails
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
2026-08-03 17:35:17 +08:00
sinohqb
0cca4963d1 refactor(tasks): unify run/campaign task registries into TaskRegistry
Both the single-run path and the campaign scheduler drove long-lived
asyncio tasks through their own duplicated _tasks/_cancel_events dicts and
shutdown loops. Collapse them into one deep TaskRegistry module,
instantiated as run_registry and campaign_registry. launch() creates the
cancel event before the task (so a cancel during startup is never lost),
wires done-callback cleanup, and is idempotent per id; this makes runs.py's
hard-cancel fallback provably dead, so it is removed. App shutdown now
gracefully stops in-flight runs too, not just campaigns.
2026-07-31 03:39:03 +08:00
sinohqb
8910fd17e0 feat(campaigns): durable scheduler loop with restart recovery and cancel
Add a thin async loop (run_campaign_loop) that ticks on real wall-clock time,
maps elapsed×time_scale to a window offset via the pure decide_schedule, spawns
due child Runs, and marks the campaign COMPLETED at window end. All authority
lives in the DB (started_at, spawned_indices, status), so the app lifespan can
resume every RUNNING campaign on startup without double-spawning and stop all
loops gracefully on shutdown. A failing plan entry is skipped and recorded
rather than wedging the campaign.

Creating a campaign now starts its loop; POST /api/campaigns/{id}/cancel stops
further spawning (completed child Runs are kept); GET /api/campaigns/{id}
reports live progress (window offset, spawned/completed Run counts).
2026-07-30 13:33:10 +08:00
sinohqb
e4404f1fa2 feat(campaigns): add Campaign persistence and create/query API
Introduce the 评估活动 (Campaign) aggregate above Run: a single-target,
service-cycle window driving a static plan. Adds Campaign/CampaignPlanEntry
models, CampaignDB table, nullable eval_runs.campaign_id, CampaignRepository,
Alembic migration, and POST/GET /api/campaigns with validation.

Ticket 01 of v0.6; no scheduling or child-run spawning yet (ADR-0003 v1).
2026-07-30 11:56:19 +08:00
sinohqb
d23b321225 fix(runs): mark orphaned running/pending runs failed on startup
Some checks failed
CI / test (push) Failing after 59s
评测任务是进程内 asyncio 任务,服务重启会中断执行且状态永远停在
running。启动时将遗留的 running/pending 运行标记为 failed(summary
写入 interrupted 错误),清理为尽力而为,不阻断启动。另将仪表盘最近
评测记录的触发方式与版本号标签位置对调。
2026-07-29 14:48:14 +08:00
sinohqb
739d586aec feat(backend): v0.4 triggered_by tracking, login gate, compare guard, dashboard stats
- EvalRun.triggered_by 全链路(manual/ai_assistant/cli)+ 迁移 b7d4e6f81c22
- 标准 agenteval-run SKILL.md 纳入版本管理,deploy 脚本同步 + API Key 注入
- 简单登录:AGENTEVAL_ADMIN_PASSWORD + HMAC 会话 token,require_auth 双凭据
- 对比报告限同场景(400)+ 空 results 误判修复
- /api/stats/dashboard 扩展聚合;/api/runs 返回场景/对象名
- 测试 218 → 232
2026-07-28 17:40:54 +08:00
sinohqb
470ff5875f feat(models): add centralized model configuration 2026-07-17 20:02:43 +08:00
sinohqb
12481cd1b8 v0.3-s1: 规则层异步化 + 工具函数去重 + HTTP 通道
## 核心变更

### 规则层全面异步化(DEBT-1)
- EvalRule.evaluate() 签名改为 async def,全量同步改造(无兼容层)
- LlmScoreRule._call_llm: requests.post → httpx.AsyncClient,彻底消除事件循环阻塞
- engine._save_rule_results: rule.evaluate() → await rule.evaluate()

### 工具函数去重(DEBT-2)
- 新建 agenteval/utils/llm.py,统一三个函数:
  - extract_reply_text (原 5 处重复)
  - extract_content_from_llm_response (原 2 处重复)
  - parse_json_from_llm_text (统一 LLM 输出 JSON 解析)
- engine.py / llm_score.py / runs.py / report.py 全部切换到 utils.llm

### HTTP 通用通道(S1-3)
- 新建 channels/http.py (HttpChannel)
  - 配置化 send_url / reply_url 模板 ({message}, {msg_id} 占位)
  - dot-path 提取 msg_id 和 reply_text
  - 可选 reply_ready_path 就绪标志
  - 长连接 AsyncClient 复用
- ChannelFactory 注册 ChannelType.HTTP → HttpChannel

### 测试
- 新增 tests/unit/test_http_channel_and_rules.py (19 个测试)
- _get_path / health_check / send / poll_reply / 超时 / 就绪标志 / async 规则评估
- 测试总数:24 → 43,全部通过

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 10:52:32 +08:00
sinohqb
a77cd83e6a v0.2.0-dev: 文件管理 + 页面布局统一 + 6 个 bug 修复
## 新增功能
- 文件管理模块:分类树 + 文件上传/下载/删除
- 文件上传支持拖拽(Dragger)+ 手动上传(customRequest 模式)

## 页面布局统一(参照评测执行页)
- 仪表盘/评测对象/评测场景/评测报告 全部改为全高 flex 布局
- 统一内联页头样式(h2 + 竖线分隔 + 描述)
- 表格撑满高度、overflow 处理
- 每页添加刷新按钮

## Bug 修复
- 分类树操作按钮 hover 不可见(CSS 规则缺失)
- 文件上传失败(multipart boundary 缺失)
- LLM API 响应 content blocks 数组格式支持(_extract_content_from_api_response)
- response_time_max_ms 被静默忽略(隐式规则传空 params)
- 空 messages 导致 IndexError 崩溃
- poll_reply 异常中止整个 run(缺 try/catch)
- engine finally 未关闭 session
- 3 个页面 UTC 时间戳解析偏差 8 小时

## 后端
- EvalEngine: poll_reply 异常保护、空 dialog 保护、session 关闭
- LLM API 响应解析支持 content-block-array 格式
- 隐式 response_time 规则正确传递 max_ms 参数

## 前端
- api.ts: 移除手动 Content-Type(让浏览器自动添加 boundary)
- Files.tsx: customRequest 替代 beforeUpload、布局优化
- index.css: 分类树 hover 规则
- Targets/Scenarios/Home/Reports: 全高布局改造
- 3 个页面时间戳改用 formatDateTime()(修复 UTC 偏差)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-16 15:25:22 +08:00