Commit Graph

7 Commits

Author SHA1 Message Date
sinohqb
f285738f6d refactor(report): split report generation from pure rendering
report.py mixed DB-reading generation with string formatting: the four
render_*_report(run_id, session) functions each re-fetched via
generate_report, so the HTML/Markdown/JSON formatting was welded to storage
and could not be unit-tested from a plain dict. Extract the formatting into a
new pure report_render module whose renderers take the already-built report
dict (no session, no storage import). Migrate every caller to generate-then-
render, delete the old coupled renderers with no back-compat shim, and drop
the _aggregate_runs middle-man alias in favour of metrics.aggregate_runs.
2026-07-31 10:19:04 +08:00
sinohqb
782916a283 refactor(metrics): type Run summary and converge cross-run aggregation
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
2026-07-31 01:57:56 +08:00
sinohqb
5db0ede4f4 refactor(judgement): converge case-pass decision into one deep module
Some checks failed
CI / test (push) Failing after 50s
「用例是否通过」此前散落 8 处且互相矛盾:engine 权威判定焊死在持久化里
不可单测;report 聚合/compare/markdown 各自从规则结果反推,规则还不一致
(markdown 用 all([]) 把故障用例误渲染成 )。

- 新增纯函数 evaluation/judgement.combine_case_outcome(RuleOutcome/
  CaseOutcome),判定组合脱离通道与 DB 可单测(判定矩阵 14 例)
- engine 调用它一次,逐用例权威结果写入 summary.case_outcomes(JSON,
  零迁移);report/compare/markdown 只读权威值,老 run fallback 反推
- 故障用例判 False(ADR-0002):修正 markdown 的  bug 与 compare 的
  None;顺带修 engine 连通用例无回复也算通过的 bug
- pass_rate 口径改为用例级(CONTEXT.md 词条),规则级保留在
  passed_rules/total_rules;CLI 对比标签同步更正
- 修 RunRepository.update 漏拷 scenario_version/triggered_by 的字段漂移
2026-07-29 19:45:02 +08:00
sinohqb
770d260750 feat(report): compare requires same scenario version (ticket 05)
Some checks failed
CI / test (push) Failing after 39s
对比报告可比性收紧为同场景同考纲版本(ADR-0001):跨版本 API 返回 400
(detail 含双方版本号),报告生成层抛 ValueError;前端对比候选按
同场景 + 同版本过滤,A 变更后自动清空不可比的 B。文档"尚未实现"标注移除。
2026-07-29 11:21:52 +08:00
sinohqb
8a526599ab feat(report): annotate connectivity cases and add judged pass rate (ticket 02)
报告层推导连通用例标记(无判定结果 + 每轮有回复 + 无用例级错误),
summary 新增 connectivity_cases 与 judged_pass_rate(无判定型用例时为 null)。
对比报告同步标注且连通用例按引擎口径计通过;总通过率口径不变(ADR-0002)。
2026-07-29 10:32:56 +08:00
sinohqb
739d586aec feat(backend): v0.4 triggered_by tracking, login gate, compare guard, dashboard stats
- EvalRun.triggered_by 全链路(manual/ai_assistant/cli)+ 迁移 b7d4e6f81c22
- 标准 agenteval-run SKILL.md 纳入版本管理,deploy 脚本同步 + API Key 注入
- 简单登录:AGENTEVAL_ADMIN_PASSWORD + HMAC 会话 token,require_auth 双凭据
- 对比报告限同场景(400)+ 空 results 误判修复
- /api/stats/dashboard 扩展聚合;/api/runs 返回场景/对象名
- 测试 218 → 232
2026-07-28 17:40:54 +08:00
sinohqb
e0b69fa2b9 v0.4-t1t2: 测试覆盖率 62%→77% + UTC 时区根本修复
## T1: P0 测试补全(+67 个测试)
- test_utils_llm.py: extract_reply_text / extract_content_from_llm_response / parse_json_from_llm_text 各边界
- test_file_repository.py: 分类 CRUD / 树形结构 / 级联删除 / 文件创建/查询/删除/物理文件清理
- test_report.py: generate_report / generate_compare_report / render_markdown / render_json
- test_llm_score.py: OpenAI 格式 / Anthropic content-block 格式 / JSON 回退解析 / 异常降级

## T2: P1 测试补全(+28 个测试)
- test_scenarios.py: 模板列表/字段完整性/规则类型有效性 + YAML/JSON 加载/校验
- test_webhook.py: 未配置不发送 / 正确 payload / secret header / 异常静默忽略
- test_reports_api.py: GET /reports/{id} / /html / /json / /markdown / /compare 集成测试

## UTC 时区根本修复
- storage/db.py: 新增 iso_utc() 函数,确保所有 datetime 序列化输出带 Z 后缀
- runs.py / files.py / report.py: 6 处 .isoformat() → iso_utc()
- 前端 toDate() 兜底仍保留(向下兼容),但后端不再输出无时区时间戳

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 14:19:16 +08:00