Commit Graph

17 Commits

Author SHA1 Message Date
sinohqb
aa40c8e0d8 refactor(campaign): type Campaign.summary as CampaignSummary VO
Some checks failed
CI / test (push) Failing after 12s
Campaign.summary was a bare Optional[dict] while RunSummary is a typed VO —
scheduler state (spawned_indices/errors) flowed untyped through
set_/get_summary. Introduce CampaignSummary + SchedulerState (extra=allow,
validate_assignment), mirroring RunSummary; campaign_runner reads/writes the
VO. Also converge the ~10 repeated JSON column get/set pairs onto
_json_dumps/_json_loads helpers, unifying ensure_ascii=False and fixing the
set_modalities ensure_ascii=True trap.
2026-07-31 15:09:10 +08:00
sinohqb
d411572607 refactor(storage): converge CRUD skeleton into BaseRepository
Target/Scenario/Run/Campaign repositories repeated the same
__init__/list_all/get/create/delete skeleton (~120 lines). Extract a
generic BaseRepository[M, DB]: subclasses declare the table + ordering
column and implement instance-method _to_db/_from_db converters (so
Scenario's _from_db can reach self.session for model bindings). Bespoke
paths (Scenario create/delete, all update) stay per-subclass. Adds
test_repository.py locking the CRUD + JSON round-trip contract.
2026-07-31 14:30:11 +08:00
sinohqb
782916a283 refactor(metrics): type Run summary and converge cross-run aggregation
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
2026-07-31 01:57:56 +08:00
sinohqb
8910fd17e0 feat(campaigns): durable scheduler loop with restart recovery and cancel
Add a thin async loop (run_campaign_loop) that ticks on real wall-clock time,
maps elapsed×time_scale to a window offset via the pure decide_schedule, spawns
due child Runs, and marks the campaign COMPLETED at window end. All authority
lives in the DB (started_at, spawned_indices, status), so the app lifespan can
resume every RUNNING campaign on startup without double-spawning and stop all
loops gracefully on shutdown. A failing plan entry is skipped and recorded
rather than wedging the campaign.

Creating a campaign now starts its loop; POST /api/campaigns/{id}/cancel stops
further spawning (completed child Runs are kept); GET /api/campaigns/{id}
reports live progress (window offset, spawned/completed Run counts).
2026-07-30 13:33:10 +08:00
sinohqb
c6b102a9b5 feat(campaigns): add scheduling decision and child-Run spawning
Add the pure scheduling seam (campaign_scheduler.decide_schedule) that, given a
static plan and window-clock offset, decides which plan entries are due and
whether the window ended — mirroring judgement.combine_case_outcome, with
time_scale confined to the clock mapping so it never touches judgement/report.

The campaign_runner shell maps injected elapsed time to a window offset, spawns
due child Runs through the existing EvalEngine.run(existing_run=...) path with
campaign_id + RunTrigger.CAMPAIGN, and persists spawned-entry indices per entry
for idempotent, restart-recoverable progress. No auto loop yet (ticket 03).
2026-07-30 12:06:31 +08:00
sinohqb
e4404f1fa2 feat(campaigns): add Campaign persistence and create/query API
Introduce the 评估活动 (Campaign) aggregate above Run: a single-target,
service-cycle window driving a static plan. Adds Campaign/CampaignPlanEntry
models, CampaignDB table, nullable eval_runs.campaign_id, CampaignRepository,
Alembic migration, and POST/GET /api/campaigns with validation.

Ticket 01 of v0.6; no scheduling or child-run spawning yet (ADR-0003 v1).
2026-07-30 11:56:19 +08:00
sinohqb
5db0ede4f4 refactor(judgement): converge case-pass decision into one deep module
Some checks failed
CI / test (push) Failing after 50s
「用例是否通过」此前散落 8 处且互相矛盾:engine 权威判定焊死在持久化里
不可单测;report 聚合/compare/markdown 各自从规则结果反推,规则还不一致
(markdown 用 all([]) 把故障用例误渲染成 )。

- 新增纯函数 evaluation/judgement.combine_case_outcome(RuleOutcome/
  CaseOutcome),判定组合脱离通道与 DB 可单测(判定矩阵 14 例)
- engine 调用它一次,逐用例权威结果写入 summary.case_outcomes(JSON,
  零迁移);report/compare/markdown 只读权威值,老 run fallback 反推
- 故障用例判 False(ADR-0002):修正 markdown 的  bug 与 compare 的
  None;顺带修 engine 连通用例无回复也算通过的 bug
- pass_rate 口径改为用例级(CONTEXT.md 词条),规则级保留在
  passed_rules/total_rules;CLI 对比标签同步更正
- 修 RunRepository.update 漏拷 scenario_version/triggered_by 的字段漂移
2026-07-29 19:45:02 +08:00
sinohqb
d23b321225 fix(runs): mark orphaned running/pending runs failed on startup
Some checks failed
CI / test (push) Failing after 59s
评测任务是进程内 asyncio 任务,服务重启会中断执行且状态永远停在
running。启动时将遗留的 running/pending 运行标记为 failed(summary
写入 interrupted 错误),清理为尽力而为,不阻断启动。另将仪表盘最近
评测记录的触发方式与版本号标签位置对调。
2026-07-29 14:48:14 +08:00
sinohqb
0a47260237 feat(run): snapshot scenario version at run creation (ticket 04)
运行创建时快照场景考纲版本,三种触发来源(手动/AI 助手/CLI)一致;
迁移回填存量运行为其场景当前版本,孤儿运行回填 1。运行列表、
报告头与对比卡片展示 v{n} 版本标签。
2026-07-29 10:59:44 +08:00
sinohqb
43e05ee38d feat(scenario): system-maintained syllabus version (ticket 03)
场景新增整型 version(迁移回填 1,batch mode)。仅考纲字段
(cases / model_bindings / llm_config)变更时升版,元数据编辑不升版,
API 传入的 version 被忽略(ADR-0001)。前端场景列表展示版本标签。
2026-07-29 10:42:35 +08:00
sinohqb
739d586aec feat(backend): v0.4 triggered_by tracking, login gate, compare guard, dashboard stats
- EvalRun.triggered_by 全链路(manual/ai_assistant/cli)+ 迁移 b7d4e6f81c22
- 标准 agenteval-run SKILL.md 纳入版本管理,deploy 脚本同步 + API Key 注入
- 简单登录:AGENTEVAL_ADMIN_PASSWORD + HMAC 会话 token,require_auth 双凭据
- 对比报告限同场景(400)+ 空 results 误判修复
- /api/stats/dashboard 扩展聚合;/api/runs 返回场景/对象名
- 测试 218 → 232
2026-07-28 17:40:54 +08:00
sinohqb
affbf60945 feat(models): add model capability metadata 2026-07-17 21:41:32 +08:00
sinohqb
470ff5875f feat(models): add centralized model configuration 2026-07-17 20:02:43 +08:00
sinohqb
d7514f4e65 refactor(files): harden storage and split frontend
Add transactional file storage workflows, typed API contracts, recursive category handling, frontend component separation, and Files API coverage.
2026-07-17 17:41:19 +08:00
sinohqb
e0b69fa2b9 v0.4-t1t2: 测试覆盖率 62%→77% + UTC 时区根本修复
## T1: P0 测试补全(+67 个测试)
- test_utils_llm.py: extract_reply_text / extract_content_from_llm_response / parse_json_from_llm_text 各边界
- test_file_repository.py: 分类 CRUD / 树形结构 / 级联删除 / 文件创建/查询/删除/物理文件清理
- test_report.py: generate_report / generate_compare_report / render_markdown / render_json
- test_llm_score.py: OpenAI 格式 / Anthropic content-block 格式 / JSON 回退解析 / 异常降级

## T2: P1 测试补全(+28 个测试)
- test_scenarios.py: 模板列表/字段完整性/规则类型有效性 + YAML/JSON 加载/校验
- test_webhook.py: 未配置不发送 / 正确 payload / secret header / 异常静默忽略
- test_reports_api.py: GET /reports/{id} / /html / /json / /markdown / /compare 集成测试

## UTC 时区根本修复
- storage/db.py: 新增 iso_utc() 函数,确保所有 datetime 序列化输出带 Z 后缀
- runs.py / files.py / report.py: 6 处 .isoformat() → iso_utc()
- 前端 toDate() 兜底仍保留(向下兼容),但后端不再输出无时区时间戳

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 14:19:16 +08:00
sinohqb
12481cd1b8 v0.3-s1: 规则层异步化 + 工具函数去重 + HTTP 通道
## 核心变更

### 规则层全面异步化(DEBT-1)
- EvalRule.evaluate() 签名改为 async def,全量同步改造(无兼容层)
- LlmScoreRule._call_llm: requests.post → httpx.AsyncClient,彻底消除事件循环阻塞
- engine._save_rule_results: rule.evaluate() → await rule.evaluate()

### 工具函数去重(DEBT-2)
- 新建 agenteval/utils/llm.py,统一三个函数:
  - extract_reply_text (原 5 处重复)
  - extract_content_from_llm_response (原 2 处重复)
  - parse_json_from_llm_text (统一 LLM 输出 JSON 解析)
- engine.py / llm_score.py / runs.py / report.py 全部切换到 utils.llm

### HTTP 通用通道(S1-3)
- 新建 channels/http.py (HttpChannel)
  - 配置化 send_url / reply_url 模板 ({message}, {msg_id} 占位)
  - dot-path 提取 msg_id 和 reply_text
  - 可选 reply_ready_path 就绪标志
  - 长连接 AsyncClient 复用
- ChannelFactory 注册 ChannelType.HTTP → HttpChannel

### 测试
- 新增 tests/unit/test_http_channel_and_rules.py (19 个测试)
- _get_path / health_check / send / poll_reply / 超时 / 就绪标志 / async 规则评估
- 测试总数:24 → 43,全部通过

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 10:52:32 +08:00
sinohqb
a77cd83e6a v0.2.0-dev: 文件管理 + 页面布局统一 + 6 个 bug 修复
## 新增功能
- 文件管理模块:分类树 + 文件上传/下载/删除
- 文件上传支持拖拽(Dragger)+ 手动上传(customRequest 模式)

## 页面布局统一(参照评测执行页)
- 仪表盘/评测对象/评测场景/评测报告 全部改为全高 flex 布局
- 统一内联页头样式(h2 + 竖线分隔 + 描述)
- 表格撑满高度、overflow 处理
- 每页添加刷新按钮

## Bug 修复
- 分类树操作按钮 hover 不可见(CSS 规则缺失)
- 文件上传失败(multipart boundary 缺失)
- LLM API 响应 content blocks 数组格式支持(_extract_content_from_api_response)
- response_time_max_ms 被静默忽略(隐式规则传空 params)
- 空 messages 导致 IndexError 崩溃
- poll_reply 异常中止整个 run(缺 try/catch)
- engine finally 未关闭 session
- 3 个页面 UTC 时间戳解析偏差 8 小时

## 后端
- EvalEngine: poll_reply 异常保护、空 dialog 保护、session 关闭
- LLM API 响应解析支持 content-block-array 格式
- 隐式 response_time 规则正确传递 max_ms 参数

## 前端
- api.ts: 移除手动 Content-Type(让浏览器自动添加 boundary)
- Files.tsx: customRequest 替代 beforeUpload、布局优化
- index.css: 分类树 hover 规则
- Targets/Scenarios/Home/Reports: 全高布局改造
- 3 个页面时间戳改用 formatDateTime()(修复 UTC 偏差)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-16 15:25:22 +08:00