Compare commits

..

No commits in common. "770d2607500d797e10ff3938441881d9591ad5a7" and "2dcf415940ad3d560d38110d02d1f131e576559f" have entirely different histories.

26 changed files with 106 additions and 772 deletions

View File

@ -4,13 +4,11 @@
**Blocked by:** None — can start immediately. **Blocked by:** None — can start immediately.
**Status:** done **Status:** ready-for-agent
- [x] 同时配置期望与规则的用例,期望不满足时用例不通过(即使显式规则全部通过) - [ ] 同时配置期望与规则的用例,期望不满足时用例不通过(即使显式规则全部通过)
- [x] 期望派生判定以 EvalResult 形式存储reason 可辨识来源为期望 - [ ] 期望派生判定以 EvalResult 形式存储reason 可辨识来源为期望
- [x] rule_logic=any/weighted 时,隐式期望判定不参与组合计算,仍作为独立硬约束 - [ ] rule_logic=any/weighted 时,隐式期望判定不参与组合计算,仍作为独立硬约束
- [x] 纯期望用例(无显式规则)的判定行为与升级前一致 - [ ] 纯期望用例(无显式规则)的判定行为与升级前一致
- [x] 引擎单元测试覆盖叠加通过/失败矩阵(先例:现有引擎测试) - [ ] 引擎单元测试覆盖叠加通过/失败矩阵(先例:现有引擎测试)
- [x] 全部现有测试保持绿色 - [ ] 全部现有测试保持绿色
> 实施备注:纯期望用例在默认 rule_logic=ALL 下行为与升级前完全一致;纯期望 + ANY/WEIGHTED 的病态组合下期望现按硬约束全判spec 决策优先于工单字面)。

View File

@ -4,10 +4,10 @@
**Blocked by:** None — can start immediately. **Blocked by:** None — can start immediately.
**Status:** done **Status:** ready-for-agent
- [x] 单次报告中连通用例带明确标记,前端可见 - [ ] 单次报告中连通用例带明确标记,前端可见
- [x] 对比报告中连通用例同样标注 - [ ] 对比报告中连通用例同样标注
- [x] summary 含连通用例数与判定型通过率;全为连通用例时判定型通过率不除零 - [ ] summary 含连通用例数与判定型通过率;全为连通用例时判定型通过率不除零
- [x] 总通过率数值与升级前一致(口径未变) - [ ] 总通过率数值与升级前一致(口径未变)
- [x] 报告单元测试覆盖标注与判定型通过率计算(先例:现有报告测试) - [ ] 报告单元测试覆盖标注与判定型通过率计算(先例:现有报告测试)

View File

@ -4,11 +4,11 @@
**Blocked by:** None — can start immediately. **Blocked by:** None — can start immediately.
**Status:** done **Status:** ready-for-agent
- [x] 新建场景 version=1数据库迁移batch mode为存量场景回填 1 - [ ] 新建场景 version=1数据库迁移batch mode为存量场景回填 1
- [x] 修改用例集 / model_bindings / llm_config 任一项后 version+1 - [ ] 修改用例集 / model_bindings / llm_config 任一项后 version+1
- [x] 仅改名称/描述/标签时 version 不变 - [ ] 仅改名称/描述/标签时 version 不变
- [x] API 请求体中携带 version 被忽略(不可外部指定) - [ ] API 请求体中携带 version 被忽略(不可外部指定)
- [x] 场景列表/详情 API 返回 version前端场景页展示 - [ ] 场景列表/详情 API 返回 version前端场景页展示
- [x] 集成测试覆盖升版与不升版两类编辑(先例:现有场景 API 测试) - [ ] 集成测试覆盖升版与不升版两类编辑(先例:现有场景 API 测试)

View File

@ -4,10 +4,10 @@
**Blocked by:** 03 — 场景版本字段与升版逻辑。 **Blocked by:** 03 — 场景版本字段与升版逻辑。
**Status:** done **Status:** ready-for-agent
- [x] 新建运行记录 scenario_version = 场景当前版本,三种触发来源一致 - [ ] 新建运行记录 scenario_version = 场景当前版本,三种触发来源一致
- [x] 迁移回填存量运行;孤儿运行(场景已删)回填 1 - [ ] 迁移回填存量运行;孤儿运行(场景已删)回填 1
- [x] 运行列表 API 与报告数据含 scenario_version - [ ] 运行列表 API 与报告数据含 scenario_version
- [x] 前端运行列表与报告头展示版本号 - [ ] 前端运行列表与报告头展示版本号
- [x] 集成测试覆盖创建快照与迁移回填(先例:现有运行 API 测试、迁移测试) - [ ] 集成测试覆盖创建快照与迁移回填(先例:现有运行 API 测试、迁移测试)

View File

@ -4,10 +4,10 @@
**Blocked by:** 04 — 运行记录场景版本。 **Blocked by:** 04 — 运行记录场景版本。
**Status:** done **Status:** ready-for-agent
- [x] 同场景不同版本的两次运行对比API 400detail 含双方版本号 - [ ] 同场景不同版本的两次运行对比API 400detail 含双方版本号
- [x] 同场景同版本对比正常生成(含动态用例场景——同考纲即可比) - [ ] 同场景同版本对比正常生成(含动态用例场景——同考纲即可比)
- [x] 报告生成层对跨版本对比抛出明确错误 - [ ] 报告生成层对跨版本对比抛出明确错误
- [x] 前端对比候选按同场景 + 同版本过滤;跨版本被拒时提示可读 - [ ] 前端对比候选按同场景 + 同版本过滤;跨版本被拒时提示可读
- [x] 集成与单元测试覆盖拒绝与放行两侧(先例:现有对比报告测试) - [ ] 集成与单元测试覆盖拒绝与放行两侧(先例:现有对比报告测试)

View File

@ -41,7 +41,7 @@ _Avoid_: 测试点、题目
_Avoid_: 差异报告 _Avoid_: 差异报告
**场景版本Scenario Version**: **场景版本Scenario Version**:
场景考纲的版本标识。仅考纲字段用例集、模型绑定、LLM 配置)变更时递增;名称、描述、标签等元数据编辑不升版。(决策见 ADR-0001 场景考纲的版本标识。仅考纲字段用例集、模型绑定、LLM 配置)变更时递增;名称、描述、标签等元数据编辑不升版。(决策见 ADR-0001,尚未实现
_Avoid_: 修订号 _Avoid_: 修订号
**评测运行Run / EvalRun**: **评测运行Run / EvalRun**:

View File

@ -118,7 +118,6 @@ class EvalEngine:
id=str(uuid.uuid4()), id=str(uuid.uuid4()),
target_id=self.target.id or "", target_id=self.target.id or "",
scenario_id=self.scenario.id or "", scenario_id=self.scenario.id or "",
scenario_version=self.scenario.version or 1,
status=RunStatus.RUNNING, status=RunStatus.RUNNING,
triggered_by=self.triggered_by, triggered_by=self.triggered_by,
started_at=utc_now(), started_at=utc_now(),
@ -380,19 +379,19 @@ class EvalEngine:
) -> tuple[bool, int, int]: ) -> tuple[bool, int, int]:
"""Apply rules and save results; returns (case_passed, passed_count, total_count). """Apply rules and save results; returns (case_passed, passed_count, total_count).
Judgement semantics (spec v0.5 / CONTEXT.md): Combination logic (case.rule_logic):
- Explicit rules are combined by case.rule_logic (ALL / ANY / WEIGHTED). ALL all rules must pass (default)
- Expectations always derive implicit checks, additive to explicit ANY at least one rule must pass
rules. They are hard constraints: they never join the rule_logic WEIGHTED weighted average score >= case.rule_pass_threshold
combination, and any implicit failure fails the case.
""" """
from agenteval.models import EvalRuleConfig from agenteval.models import EvalRuleConfig
rules_config: list[EvalRuleConfig] = list(case.eval_rules) rules_config: list[EvalRuleConfig] = list(case.eval_rules)
implicit_config: list[EvalRuleConfig] = [] # If no explicit rules, derive implicit rules from expectations.
if not rules_config:
if case.expectations.response_time_max_ms: if case.expectations.response_time_max_ms:
implicit_config.append( rules_config.append(
EvalRuleConfig( EvalRuleConfig(
type="response_time", type="response_time",
params={ params={
@ -401,7 +400,7 @@ class EvalEngine:
) )
) )
if case.expectations.keywords_include or case.expectations.keywords_exclude: if case.expectations.keywords_include or case.expectations.keywords_exclude:
implicit_config.append( rules_config.append(
EvalRuleConfig( EvalRuleConfig(
type="keyword_match", type="keyword_match",
params={ params={
@ -411,20 +410,16 @@ class EvalEngine:
) )
) )
if not rules_config and not implicit_config: if not rules_config:
# 连通用例:无任何判定标准,收到回复即通过 # No rules defined and no expectations → case passes with no checks
return True, 0, 0 return True, 0, 0
passed_count = 0 passed_count = 0
total_count = 0 total_count = 0
explicit_passed = 0
explicit_total = 0
weighted_score = 0.0 weighted_score = 0.0
total_weight = 0.0 total_weight = 0.0
implicit_all_passed = True
all_rules = [(cfg, False) for cfg in rules_config] + [(cfg, True) for cfg in implicit_config] for rule_config in rules_config:
for rule_config, is_implicit in all_rules:
purpose = { purpose = {
"llm_score": ModelPurpose.JUDGE, "llm_score": ModelPurpose.JUDGE,
"semantic_similarity": ModelPurpose.EMBEDDING, "semantic_similarity": ModelPurpose.EMBEDDING,
@ -441,7 +436,6 @@ class EvalEngine:
result = await rule.evaluate(case, dialog) result = await rule.evaluate(case, dialog)
except Exception as exc: except Exception as exc:
result = RuleResult(passed=False, reason=f"模型配置解析失败: {exc}") result = RuleResult(passed=False, reason=f"模型配置解析失败: {exc}")
reason = f"[期望] {result.reason}" if is_implicit else result.reason
eval_result = EvalResult( eval_result = EvalResult(
id=str(uuid.uuid4()), id=str(uuid.uuid4()),
run_id=run.id, run_id=run.id,
@ -450,20 +444,13 @@ class EvalEngine:
rule_type=rule_config.type, rule_type=rule_config.type,
passed=result.passed, passed=result.passed,
score=result.score, score=result.score,
reason=reason, reason=result.reason,
) )
self.result_repo.save_result(eval_result) self.result_repo.save_result(eval_result)
total_count += 1 total_count += 1
if result.passed: if result.passed:
passed_count += 1 passed_count += 1
if is_implicit:
if not result.passed:
implicit_all_passed = False
else:
explicit_total += 1
if result.passed:
explicit_passed += 1
# Weighted scoring: use rule score (default 1.0 if passed, 0.0 if failed) # Weighted scoring: use rule score (default 1.0 if passed, 0.0 if failed)
score_val = result.score if result.score is not None else (1.0 if result.passed else 0.0) score_val = result.score if result.score is not None else (1.0 if result.passed else 0.0)
weight = rule_config.weight weight = rule_config.weight
@ -479,24 +466,22 @@ class EvalEngine:
"rule_type": rule_config.type, "rule_type": rule_config.type,
"passed": result.passed, "passed": result.passed,
"score": result.score, "score": result.score,
"reason": reason, "reason": result.reason,
"weight": rule_config.weight, "weight": weight,
}, },
) )
# Combine explicit rules by rule_logic; no explicit rules → vacuously true. # Determine case pass/fail based on rule_logic
logic = case.rule_logic logic = case.rule_logic
if not rules_config: if logic == RuleLogic.ALL:
explicit_ok = True case_passed = passed_count == total_count
elif logic == RuleLogic.ANY: elif logic == RuleLogic.ANY:
explicit_ok = explicit_passed > 0 case_passed = passed_count > 0
elif logic == RuleLogic.WEIGHTED: elif logic == RuleLogic.WEIGHTED:
avg = weighted_score / total_weight if total_weight > 0 else 0.0 avg = weighted_score / total_weight if total_weight > 0 else 0.0
explicit_ok = avg >= case.rule_pass_threshold case_passed = avg >= case.rule_pass_threshold
else: # ALL and fallback else:
explicit_ok = explicit_passed == explicit_total case_passed = passed_count == total_count
case_passed = explicit_ok and implicit_all_passed
return case_passed, passed_count, total_count return case_passed, passed_count, total_count

View File

@ -111,11 +111,9 @@ def generate_report(run_id: str, session=None) -> dict[str, Any]:
# Group by case # Group by case
case_map: dict[str, dict[str, Any]] = {} case_map: dict[str, dict[str, Any]] = {}
for turn in turns: for turn in turns:
case_map.setdefault(turn.case_id, {"turns": [], "results": [], "all_replied": True}) case_map.setdefault(turn.case_id, {"turns": [], "results": []})
sent = turn.get_sent_message() sent = turn.get_sent_message()
reply = turn.get_reply() reply = turn.get_reply()
if reply is None:
case_map[turn.case_id]["all_replied"] = False
case_map[turn.case_id]["turns"].append( case_map[turn.case_id]["turns"].append(
{ {
"round": turn.round_index, "round": turn.round_index,
@ -127,7 +125,7 @@ def generate_report(run_id: str, session=None) -> dict[str, Any]:
) )
for result in results: for result in results:
case_map.setdefault(result.case_id, {"turns": [], "results": [], "all_replied": True}) case_map.setdefault(result.case_id, {"turns": [], "results": []})
case_map[result.case_id]["results"].append( case_map[result.case_id]["results"].append(
{ {
"rule_type": result.rule_type, "rule_type": result.rule_type,
@ -137,54 +135,34 @@ def generate_report(run_id: str, session=None) -> dict[str, Any]:
} }
) )
summary = run.summary or {}
errored_case_ids = {e.get("case_id") for e in summary.get("case_errors", [])}
cases = [] cases = []
for case_id in sorted(case_map.keys()): for case_id in sorted(case_map.keys()):
item = case_map[case_id] item = case_map[case_id]
# 连通用例无任何判定结果且每轮都收到回复、无用例级错误CONTEXT.md
connectivity = (
not item["results"]
and bool(item["turns"])
and item["all_replied"]
and case_id not in errored_case_ids
)
cases.append( cases.append(
{ {
"case_id": case_id, "case_id": case_id,
"connectivity": connectivity,
"turns": sorted(item["turns"], key=lambda x: x["round"]), "turns": sorted(item["turns"], key=lambda x: x["round"]),
"results": item["results"], "results": item["results"],
} }
) )
total_cases = summary.get("total_cases", 0) summary = run.summary or {}
passed_cases = summary.get("passed_cases", 0)
connectivity_count = sum(1 for c in cases if c["connectivity"])
judged_total = total_cases - connectivity_count
# 连通用例按引擎口径计通过,判定型通过数 = 总通过数 - 连通用例数
judged_pass_rate = round((passed_cases - connectivity_count) / judged_total, 4) if judged_total > 0 else None
return { return {
"run_id": run.id, "run_id": run.id,
"target_id": run.target_id, "target_id": run.target_id,
"target_name": target.name if target else "未知", "target_name": target.name if target else "未知",
"scenario_id": run.scenario_id, "scenario_id": run.scenario_id,
"scenario_name": scenario.name if scenario else "未知", "scenario_name": scenario.name if scenario else "未知",
"scenario_version": run.scenario_version,
"status": run.status.value, "status": run.status.value,
"started_at": iso_utc(run.started_at), "started_at": iso_utc(run.started_at),
"completed_at": iso_utc(run.completed_at), "completed_at": iso_utc(run.completed_at),
"summary": { "summary": {
"total_cases": total_cases, "total_cases": summary.get("total_cases", 0),
"passed_cases": passed_cases, "passed_cases": summary.get("passed_cases", 0),
"failed_cases": summary.get("failed_cases", 0), "failed_cases": summary.get("failed_cases", 0),
"total_rules": summary.get("total_rules", 0), "total_rules": summary.get("total_rules", 0),
"passed_rules": summary.get("passed_rules", 0), "passed_rules": summary.get("passed_rules", 0),
"pass_rate": summary.get("pass_rate", 0.0), "pass_rate": summary.get("pass_rate", 0.0),
"connectivity_cases": connectivity_count,
"judged_pass_rate": judged_pass_rate,
}, },
"cases": cases, "cases": cases,
} }
@ -199,12 +177,6 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
# "changed" and the diff would be meaningless — reject early. # "changed" and the diff would be meaningless — reject early.
if report_a.get("scenario_id") != report_b.get("scenario_id"): if report_a.get("scenario_id") != report_b.get("scenario_id"):
raise ValueError("compare report requires both runs to use the same scenario") raise ValueError("compare report requires both runs to use the same scenario")
# 同场景还须同考纲版本才可比ADR-0001
if report_a.get("scenario_version") != report_b.get("scenario_version"):
raise ValueError(
"compare report requires the same scenario version "
f"(A: v{report_a.get('scenario_version')}, B: v{report_b.get('scenario_version')})"
)
def _summary_delta(key: str) -> float: def _summary_delta(key: str) -> float:
return report_b["summary"][key] - report_a["summary"][key] return report_b["summary"][key] - report_a["summary"][key]
@ -222,9 +194,6 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
def _case_passed(c): def _case_passed(c):
if not c: if not c:
return None return None
if c.get("connectivity"):
# 连通用例收到回复即通过(引擎口径)
return True
results = c.get("results", []) results = c.get("results", [])
if not results: if not results:
# No rule results (e.g. errored case) must not count as passed. # No rule results (e.g. errored case) must not count as passed.
@ -234,7 +203,6 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
case_diffs.append( case_diffs.append(
{ {
"case_id": cid, "case_id": cid,
"connectivity": bool((ca and ca.get("connectivity")) or (cb and cb.get("connectivity"))),
"run_a_passed": _case_passed(ca), "run_a_passed": _case_passed(ca),
"run_b_passed": _case_passed(cb), "run_b_passed": _case_passed(cb),
"changed": _case_passed(ca) != _case_passed(cb), "changed": _case_passed(ca) != _case_passed(cb),
@ -248,7 +216,6 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
"run_id": run_id_1, "run_id": run_id_1,
"target_name": report_a.get("target_name"), "target_name": report_a.get("target_name"),
"scenario_name": report_a.get("scenario_name"), "scenario_name": report_a.get("scenario_name"),
"scenario_version": report_a.get("scenario_version"),
"status": report_a.get("status"), "status": report_a.get("status"),
"started_at": report_a.get("started_at"), "started_at": report_a.get("started_at"),
"summary": report_a["summary"], "summary": report_a["summary"],
@ -257,7 +224,6 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
"run_id": run_id_2, "run_id": run_id_2,
"target_name": report_b.get("target_name"), "target_name": report_b.get("target_name"),
"scenario_name": report_b.get("scenario_name"), "scenario_name": report_b.get("scenario_name"),
"scenario_version": report_b.get("scenario_version"),
"status": report_b.get("status"), "status": report_b.get("status"),
"started_at": report_b.get("started_at"), "started_at": report_b.get("started_at"),
"summary": report_b["summary"], "summary": report_b["summary"],
@ -276,8 +242,6 @@ def render_markdown_report(run_id: str, session=None) -> str:
"""Render a report as Markdown string.""" """Render a report as Markdown string."""
report = generate_report(run_id, session) report = generate_report(run_id, session)
s = report["summary"] s = report["summary"]
judged_rate = s.get("judged_pass_rate")
judged_rate_text = "" if judged_rate is None else f"{judged_rate * 100:.1f}%"
lines: list[str] = [ lines: list[str] = [
f"# 评测报告 — {report.get('scenario_name', run_id)}", f"# 评测报告 — {report.get('scenario_name', run_id)}",
"", "",
@ -297,23 +261,15 @@ def render_markdown_report(run_id: str, session=None) -> str:
f"| 总规则数 | {s['total_rules']} |", f"| 总规则数 | {s['total_rules']} |",
f"| 通过规则 | {s['passed_rules']} |", f"| 通过规则 | {s['passed_rules']} |",
f"| 通过率 | {s['pass_rate'] * 100:.1f}% |", f"| 通过率 | {s['pass_rate'] * 100:.1f}% |",
f"| 连通用例 | {s.get('connectivity_cases', 0)} |",
f"| 判定型通过率 | {judged_rate_text} |",
"", "",
"## 用例明细", "## 用例明细",
"", "",
] ]
for case in report.get("cases", []): for case in report.get("cases", []):
if case.get("connectivity"):
badge = "🔗"
else:
case_passed = all(r["passed"] for r in case.get("results", [])) case_passed = all(r["passed"] for r in case.get("results", []))
badge = "" if case_passed else "" badge = "" if case_passed else ""
title = f"### {badge} 用例 `{case['case_id']}`" lines.append(f"### {badge} 用例 `{case['case_id']}`")
if case.get("connectivity"):
title += "(连通用例,未配置判定标准)"
lines.append(title)
lines.append("") lines.append("")
for turn in case.get("turns", []): for turn in case.get("turns", []):

View File

@ -130,8 +130,6 @@ class Scenario(BaseModel):
cases: list[Case] = Field(default_factory=list) cases: list[Case] = Field(default_factory=list)
model_bindings: dict[ModelPurpose, str] = Field(default_factory=dict) model_bindings: dict[ModelPurpose, str] = Field(default_factory=dict)
llm_config: Optional[dict[str, Any]] = None llm_config: Optional[dict[str, Any]] = None
# 考纲版本由系统维护ADR-0001API 传入值会被忽略
version: int = 1
created_at: Optional[datetime] = None created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None updated_at: Optional[datetime] = None
@ -162,8 +160,6 @@ class EvalRun(BaseModel):
id: Optional[str] = None id: Optional[str] = None
target_id: str target_id: str
scenario_id: str scenario_id: str
# 创建时快照的场景考纲版本ADR-0001
scenario_version: int = 1
status: RunStatus = RunStatus.PENDING status: RunStatus = RunStatus.PENDING
triggered_by: RunTrigger = RunTrigger.MANUAL triggered_by: RunTrigger = RunTrigger.MANUAL
started_at: Optional[datetime] = None started_at: Optional[datetime] = None

View File

@ -82,7 +82,6 @@ class ScenarioDB(SQLModel, table=True):
tags: str = "[]" tags: str = "[]"
cases: str = "[]" cases: str = "[]"
llm_config: Optional[str] = None llm_config: Optional[str] = None
version: int = Field(default=1)
created_at: Optional[datetime] = Field(default_factory=utc_now) created_at: Optional[datetime] = Field(default_factory=utc_now)
updated_at: Optional[datetime] = Field(default_factory=utc_now) updated_at: Optional[datetime] = Field(default_factory=utc_now)
@ -168,7 +167,6 @@ class EvalRunDB(SQLModel, table=True):
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True) id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
target_id: Optional[str] = Field(default=None, foreign_key="eval_targets.id") target_id: Optional[str] = Field(default=None, foreign_key="eval_targets.id")
scenario_id: Optional[str] = Field(default=None, foreign_key="scenarios.id") scenario_id: Optional[str] = Field(default=None, foreign_key="scenarios.id")
scenario_version: int = Field(default=1)
status: str = "pending" status: str = "pending"
triggered_by: str = Field(default="manual") triggered_by: str = Field(default="manual")
started_at: Optional[datetime] = Field(default_factory=utc_now) started_at: Optional[datetime] = Field(default_factory=utc_now)

View File

@ -56,8 +56,7 @@ def _scenario_to_db(scenario: Scenario) -> ScenarioDB:
updated_at=scenario.updated_at or utc_now(), updated_at=scenario.updated_at or utc_now(),
) )
db.set_tags(scenario.tags) db.set_tags(scenario.tags)
# mode="json" 与 update() 的考纲比较保持同一序列化形态,避免假升版 db.set_cases([case.model_dump() for case in scenario.cases])
db.set_cases([case.model_dump(mode="json") for case in scenario.cases])
db.set_llm_config(scenario.llm_config) db.set_llm_config(scenario.llm_config)
return db return db
@ -72,7 +71,6 @@ def _scenario_from_db(db: ScenarioDB, session: Session) -> Scenario:
cases=[Case(**case) for case in db.get_cases()], cases=[Case(**case) for case in db.get_cases()],
model_bindings=bindings, model_bindings=bindings,
llm_config=db.get_llm_config(), llm_config=db.get_llm_config(),
version=db.version or 1,
created_at=db.created_at, created_at=db.created_at,
updated_at=db.updated_at, updated_at=db.updated_at,
) )
@ -83,7 +81,6 @@ def _run_to_db(run: EvalRun) -> EvalRunDB:
id=run.id, id=run.id,
target_id=run.target_id, target_id=run.target_id,
scenario_id=run.scenario_id, scenario_id=run.scenario_id,
scenario_version=run.scenario_version,
status=run.status.value, status=run.status.value,
triggered_by=run.triggered_by.value, triggered_by=run.triggered_by.value,
started_at=run.started_at, started_at=run.started_at,
@ -99,7 +96,6 @@ def _run_from_db(db: EvalRunDB) -> EvalRun:
id=db.id, id=db.id,
target_id=db.target_id, target_id=db.target_id,
scenario_id=db.scenario_id, scenario_id=db.scenario_id,
scenario_version=db.scenario_version or 1,
status=db.status, status=db.status,
triggered_by=db.triggered_by or "manual", triggered_by=db.triggered_by or "manual",
started_at=db.started_at, started_at=db.started_at,
@ -216,21 +212,10 @@ class ScenarioRepository:
bindings = {purpose.value: config_id for purpose, config_id in scenario.model_bindings.items()} bindings = {purpose.value: config_id for purpose, config_id in scenario.model_bindings.items()}
try: try:
ModelConfigService(self.session).validate_bindings(bindings) ModelConfigService(self.session).validate_bindings(bindings)
# 考纲字段cases / model_bindings / llm_config变更才升版ADR-0001
# 版本由系统维护,忽略 scenario.version 的外部传入值。
new_cases = [case.model_dump(mode="json") for case in scenario.cases]
old_bindings = ScenarioModelBindingRepository(self.session).get_for_scenario(existing.id or "")
syllabus_changed = (
existing.get_cases() != new_cases
or existing.get_llm_config() != scenario.llm_config
or old_bindings != bindings
)
if syllabus_changed:
existing.version = (existing.version or 1) + 1
existing.name = scenario.name existing.name = scenario.name
existing.description = scenario.description existing.description = scenario.description
existing.set_tags(scenario.tags) existing.set_tags(scenario.tags)
existing.set_cases(new_cases) existing.set_cases([case.model_dump() for case in scenario.cases])
existing.set_llm_config(scenario.llm_config) existing.set_llm_config(scenario.llm_config)
existing.updated_at = utc_now() existing.updated_at = utc_now()
self.session.add(existing) self.session.add(existing)

View File

@ -31,12 +31,6 @@ def get_compare_report(
raise HTTPException(status_code=404, detail=f"run not found: {run2}") raise HTTPException(status_code=404, detail=f"run not found: {run2}")
if run_a.scenario_id != run_b.scenario_id: if run_a.scenario_id != run_b.scenario_id:
raise HTTPException(status_code=400, detail="对比报告要求两个运行使用相同场景") raise HTTPException(status_code=400, detail="对比报告要求两个运行使用相同场景")
if run_a.scenario_version != run_b.scenario_version:
raise HTTPException(
status_code=400,
detail=f"对比报告要求相同的场景考纲版本A: v{run_a.scenario_version}, B: v{run_b.scenario_version}"
"考纲变更前后的运行不可比",
)
return generate_compare_report(run1, run2, session) return generate_compare_report(run1, run2, session)

View File

@ -95,7 +95,6 @@ async def start_run(
run = EvalRun( run = EvalRun(
target_id=request.target_id, target_id=request.target_id,
scenario_id=request.scenario_id, scenario_id=request.scenario_id,
scenario_version=scenario.version or 1,
triggered_by=request.triggered_by, triggered_by=request.triggered_by,
) )
run = RunRepository(session).create(run) run = RunRepository(session).create(run)

View File

@ -10,5 +10,5 @@
## Consequences ## Consequences
- 数据模型已加 scenario version 字段Run 创建时快照所用版本v0.5 已实现) - 数据模型需加 scenario version 字段Run 需记录所用版本v0.5 实施项,尚未实现)
- 动态用例每次运行题目不同不影响可比性——可比性单位是"同考纲"(同场景同版本),不是"同考卷" - 动态用例每次运行题目不同不影响可比性——可比性单位是"同考纲"(同场景同版本),不是"同考卷"

View File

@ -70,7 +70,6 @@ export interface Scenario {
tags: string[] tags: string[]
cases: any[] cases: any[]
model_bindings: Record<string, string> model_bindings: Record<string, string>
version: number
created_at: string created_at: string
updated_at: string updated_at: string
} }
@ -141,7 +140,6 @@ export interface Run {
id: string id: string
target_id: string target_id: string
scenario_id: string scenario_id: string
scenario_version?: number
status: string status: string
triggered_by?: RunTrigger triggered_by?: RunTrigger
scenario_name?: string | null scenario_name?: string | null

View File

@ -292,13 +292,6 @@ function RunRow({ r, selected, targetName, scenarioName, onSelect, onOpenReport,
<span>{shortDateTime(r.started_at)}</span> <span>{shortDateTime(r.started_at)}</span>
<span>·</span> <span>·</span>
<span>{elapsedStr(r.started_at, r.completed_at)}</span> <span>{elapsedStr(r.started_at, r.completed_at)}</span>
{r.scenario_version != null && (
<Tooltip title="场景考纲版本">
<Tag style={{ marginLeft: 2, marginRight: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}>
v{r.scenario_version}
</Tag>
</Tooltip>
)}
{r.triggered_by && r.triggered_by !== 'manual' && ( {r.triggered_by && r.triggered_by !== 'manual' && (
<Tag <Tag
color={triggerColors[r.triggered_by] ?? 'default'} color={triggerColors[r.triggered_by] ?? 'default'}

View File

@ -1,7 +1,7 @@
import { useEffect, useMemo, useState } from 'react' import { useEffect, useMemo, useState } from 'react'
import { useSearchParams } from 'react-router-dom' import { useSearchParams } from 'react-router-dom'
import { import {
Alert, Button, Card, Col, Collapse, Descriptions, Empty, Row, Segmented, Button, Card, Col, Collapse, Descriptions, Empty, Row, Segmented,
Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge, message, Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge, message,
} from 'antd' } from 'antd'
import { import {
@ -29,7 +29,6 @@ interface RuleResultData {
interface CaseReport { interface CaseReport {
case_id: string case_id: string
connectivity: boolean
turns: TurnData[] turns: TurnData[]
results: RuleResultData[] results: RuleResultData[]
} }
@ -38,7 +37,6 @@ interface Report {
run_id: string run_id: string
target_name: string target_name: string
scenario_name: string scenario_name: string
scenario_version?: number
status: string status: string
started_at: string started_at: string
completed_at: string | null completed_at: string | null
@ -49,19 +47,16 @@ interface Report {
total_rules: number total_rules: number
passed_rules: number passed_rules: number
pass_rate: number pass_rate: number
connectivity_cases: number
judged_pass_rate: number | null
} }
cases: CaseReport[] cases: CaseReport[]
} }
interface CompareResult { interface CompareResult {
run_a: { run_id: string; target_name: string; scenario_name: string; scenario_version?: number; status: string; started_at: string; summary: Report['summary'] } run_a: { run_id: string; target_name: string; scenario_name: string; status: string; started_at: string; summary: Report['summary'] }
run_b: { run_id: string; target_name: string; scenario_name: string; scenario_version?: number; status: string; started_at: string; summary: Report['summary'] } run_b: { run_id: string; target_name: string; scenario_name: string; status: string; started_at: string; summary: Report['summary'] }
delta: { pass_rate: number; passed_cases: number; passed_rules: number } delta: { pass_rate: number; passed_cases: number; passed_rules: number }
cases: Array<{ cases: Array<{
case_id: string case_id: string
connectivity: boolean
run_a_passed: boolean | null run_a_passed: boolean | null
run_b_passed: boolean | null run_b_passed: boolean | null
changed: boolean changed: boolean
@ -125,11 +120,10 @@ export default function ReportsPage() {
const handleView = (runId: string) => { const handleView = (runId: string) => {
setSearchParams({ run: runId }) setSearchParams({ run: runId })
loadReport(runId) loadReport(runId)
// 对比报告要求同场景同版本A 变更后若 B 不可比则清空 // 对比报告要求同场景A 变更后若 B 场景不同则清空
const a = runs.find((r) => r.id === runId) const a = runs.find((r) => r.id === runId)
const b = runs.find((r) => r.id === compareRunId) const b = runs.find((r) => r.id === compareRunId)
if (a && b && (a.scenario_id !== b.scenario_id if (a && b && a.scenario_id !== b.scenario_id) setCompareRunId('')
|| (a.scenario_version ?? 1) !== (b.scenario_version ?? 1))) setCompareRunId('')
} }
const handleCompare = async () => { const handleCompare = async () => {
@ -206,14 +200,10 @@ export default function ReportsPage() {
const runSelectOptions = filteredRuns.map(buildOption) const runSelectOptions = filteredRuns.map(buildOption)
// 报告 B 只能选与报告 A 同场景同版本(同考纲)的 run // 报告 B 只能选与报告 A 同场景的 run
const selectedRun = runs.find((r) => r.id === selectedRunId) const selectedRun = runs.find((r) => r.id === selectedRunId)
const compareOptions = runs const compareOptions = runs
.filter((r) => .filter((r) => r.id !== selectedRunId && selectedRun && r.scenario_id === selectedRun.scenario_id)
r.id !== selectedRunId
&& selectedRun
&& r.scenario_id === selectedRun.scenario_id
&& (r.scenario_version ?? 1) === (selectedRun.scenario_version ?? 1))
.map(buildOption) .map(buildOption)
const optionFilter = (input: string, opt?: { searchText?: string }) => const optionFilter = (input: string, opt?: { searchText?: string }) =>
@ -296,7 +286,7 @@ export default function ReportsPage() {
filterOption={optionFilter} filterOption={optionFilter}
options={compareOptions} options={compareOptions}
popupMatchSelectWidth={false} popupMatchSelectWidth={false}
notFoundContent={<Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="没有同场景同版本的其他评测记录" />} notFoundContent={<Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="没有同场景的其他评测记录" />}
/> />
</Tooltip> </Tooltip>
<Button <Button
@ -374,23 +364,10 @@ function SingleReportView({ report }: { report: Report | null }) {
</Col> </Col>
</Row> </Row>
{report.summary.connectivity_cases > 0 && (
<Alert type="info" showIcon style={{ marginBottom: 16 }}
message={`本次运行含 ${report.summary.connectivity_cases} 个连通用例(未配置判定标准,收到回复即通过)` +
(report.summary.judged_pass_rate != null
? `,判定型通过率 ${(report.summary.judged_pass_rate * 100).toFixed(1)}%`
: ',无判定型用例')} />
)}
<Card style={{ marginBottom: 16 }}> <Card style={{ marginBottom: 16 }}>
<Descriptions size="small" column={2}> <Descriptions size="small" column={2}>
<Descriptions.Item label="评测对象">{report.target_name}</Descriptions.Item> <Descriptions.Item label="评测对象">{report.target_name}</Descriptions.Item>
<Descriptions.Item label="评测场景"> <Descriptions.Item label="评测场景">{report.scenario_name}</Descriptions.Item>
<Space size={6}>
{report.scenario_name}
{report.scenario_version != null && <Tag color="geekblue">v{report.scenario_version}</Tag>}
</Space>
</Descriptions.Item>
<Descriptions.Item label="开始时间">{formatDateTime(report.started_at)}</Descriptions.Item> <Descriptions.Item label="开始时间">{formatDateTime(report.started_at)}</Descriptions.Item>
<Descriptions.Item label="完成时间">{report.completed_at ? formatDateTime(report.completed_at) : '-'}</Descriptions.Item> <Descriptions.Item label="完成时间">{report.completed_at ? formatDateTime(report.completed_at) : '-'}</Descriptions.Item>
</Descriptions> </Descriptions>
@ -402,9 +379,7 @@ function SingleReportView({ report }: { report: Report | null }) {
label: ( label: (
<Space> <Space>
<span style={{ fontWeight: 500 }}>{c.case_id}</span> <span style={{ fontWeight: 500 }}>{c.case_id}</span>
{c.connectivity {c.results.every((r) => r.passed)
? <Tag color="blue"></Tag>
: c.results.every((r) => r.passed)
? <Tag color="success"></Tag> ? <Tag color="success"></Tag>
: <Tag color="error"></Tag>} : <Tag color="error"></Tag>}
<Tag>{c.turns.length} </Tag> <Tag>{c.turns.length} </Tag>
@ -485,10 +460,7 @@ function CompareView({ result }: { result: CompareResult | null }) {
<Col span={11}> <Col span={11}>
<Card title={<span style={{ color: '#1677ff' }}> A {run_a.run_id.slice(0, 8)}</span>} size="small"> <Card title={<span style={{ color: '#1677ff' }}> A {run_a.run_id.slice(0, 8)}</span>} size="small">
<Descriptions size="small" column={1}> <Descriptions size="small" column={1}>
<Descriptions.Item label="场景"> <Descriptions.Item label="场景">{run_a.scenario_name}</Descriptions.Item>
{run_a.scenario_name}
{run_a.scenario_version != null && <Tag color="geekblue" style={{ marginLeft: 6 }}>v{run_a.scenario_version}</Tag>}
</Descriptions.Item>
<Descriptions.Item label="时间">{formatDateTime(run_a.started_at)}</Descriptions.Item> <Descriptions.Item label="时间">{formatDateTime(run_a.started_at)}</Descriptions.Item>
<Descriptions.Item label="通过率">{(run_a.summary.pass_rate * 100).toFixed(1)}%</Descriptions.Item> <Descriptions.Item label="通过率">{(run_a.summary.pass_rate * 100).toFixed(1)}%</Descriptions.Item>
<Descriptions.Item label="用例">{run_a.summary.passed_cases}/{run_a.summary.total_cases}</Descriptions.Item> <Descriptions.Item label="用例">{run_a.summary.passed_cases}/{run_a.summary.total_cases}</Descriptions.Item>
@ -506,10 +478,7 @@ function CompareView({ result }: { result: CompareResult | null }) {
<Col span={11}> <Col span={11}>
<Card title={<span style={{ color: '#52c41a' }}> B {run_b.run_id.slice(0, 8)}</span>} size="small"> <Card title={<span style={{ color: '#52c41a' }}> B {run_b.run_id.slice(0, 8)}</span>} size="small">
<Descriptions size="small" column={1}> <Descriptions size="small" column={1}>
<Descriptions.Item label="场景"> <Descriptions.Item label="场景">{run_b.scenario_name}</Descriptions.Item>
{run_b.scenario_name}
{run_b.scenario_version != null && <Tag color="geekblue" style={{ marginLeft: 6 }}>v{run_b.scenario_version}</Tag>}
</Descriptions.Item>
<Descriptions.Item label="时间">{formatDateTime(run_b.started_at)}</Descriptions.Item> <Descriptions.Item label="时间">{formatDateTime(run_b.started_at)}</Descriptions.Item>
<Descriptions.Item label="通过率">{(run_b.summary.pass_rate * 100).toFixed(1)}%</Descriptions.Item> <Descriptions.Item label="通过率">{(run_b.summary.pass_rate * 100).toFixed(1)}%</Descriptions.Item>
<Descriptions.Item label="用例">{run_b.summary.passed_cases}/{run_b.summary.total_cases}</Descriptions.Item> <Descriptions.Item label="用例">{run_b.summary.passed_cases}/{run_b.summary.total_cases}</Descriptions.Item>
@ -532,7 +501,6 @@ function CompareView({ result }: { result: CompareResult | null }) {
<Space> <Space>
{r.changed && <Badge dot color="orange" />} {r.changed && <Badge dot color="orange" />}
<span style={{ fontWeight: r.changed ? 600 : 400 }}>{id}</span> <span style={{ fontWeight: r.changed ? 600 : 400 }}>{id}</span>
{r.connectivity && <Tag color="blue"></Tag>}
</Space> </Space>
), ),
}, },

View File

@ -175,9 +175,6 @@ export default function ScenariosPage() {
{ title: '名称', dataIndex: 'name', key: 'name', width: 200, { title: '名称', dataIndex: 'name', key: 'name', width: 200,
render: (v: string) => <span style={{ fontWeight: 500 }}>{v}</span>, render: (v: string) => <span style={{ fontWeight: 500 }}>{v}</span>,
}, },
{ title: '版本', dataIndex: 'version', key: 'version', width: 70,
render: (v: number) => <Tag color="geekblue">v{v ?? 1}</Tag>,
},
{ title: '描述', dataIndex: 'description', key: 'description', ellipsis: true }, { title: '描述', dataIndex: 'description', key: 'description', ellipsis: true },
{ title: '标签', dataIndex: 'tags', key: 'tags', width: 200, { title: '标签', dataIndex: 'tags', key: 'tags', width: 200,
render: (tags: string[]) => tags.map((t) => <Tag key={t}>{t}</Tag>), render: (tags: string[]) => tags.map((t) => <Tag key={t}>{t}</Tag>),

View File

@ -1,29 +0,0 @@
"""add version to scenarios
Revision ID: c8e2f5a7b901
Revises: b7d4e6f81c22
Create Date: 2026-07-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "c8e2f5a7b901"
down_revision: Union[str, Sequence[str], None] = "b7d4e6f81c22"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("scenarios") as batch_op:
batch_op.add_column(
sa.Column("version", sa.Integer(), nullable=False, server_default="1")
)
def downgrade() -> None:
with op.batch_alter_table("scenarios") as batch_op:
batch_op.drop_column("version")

View File

@ -1,34 +0,0 @@
"""add scenario_version to eval_runs
Revision ID: d5b8c2e4f617
Revises: c8e2f5a7b901
Create Date: 2026-07-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "d5b8c2e4f617"
down_revision: Union[str, Sequence[str], None] = "c8e2f5a7b901"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.add_column(
sa.Column("scenario_version", sa.Integer(), nullable=False, server_default="1")
)
# 存量运行回填其场景当前版本;孤儿运行(场景已删)保持默认 1
op.execute(
"UPDATE eval_runs SET scenario_version = COALESCE("
"(SELECT version FROM scenarios WHERE scenarios.id = eval_runs.scenario_id), 1)"
)
def downgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.drop_column("scenario_version")

View File

@ -39,12 +39,7 @@ def client_with_db(tmp_path):
session.close() session.close()
def _seed_run( def _seed_run(session: Session, name_suffix: str = "", scenario_id: str | None = None) -> str:
session: Session,
name_suffix: str = "",
scenario_id: str | None = None,
scenario_version: int = 1,
) -> str:
target = EvalTarget( target = EvalTarget(
name=f"target{name_suffix}", name=f"target{name_suffix}",
platform=PlatformType.AI_DIGITAL_EMPLOYEE, platform=PlatformType.AI_DIGITAL_EMPLOYEE,
@ -65,7 +60,6 @@ def _seed_run(
run = EvalRun( run = EvalRun(
target_id=target.id, target_id=target.id,
scenario_id=scenario_id, scenario_id=scenario_id,
scenario_version=scenario_version,
status=RunStatus.COMPLETED, status=RunStatus.COMPLETED,
) )
run = RunRepository(session).create(run) run = RunRepository(session).create(run)
@ -171,29 +165,6 @@ def test_compare_report_different_scenarios_400(client_with_db):
assert "相同场景" in resp.json()["detail"] assert "相同场景" in resp.json()["detail"]
def test_compare_report_cross_version_400(client_with_db):
"""同场景不同考纲版本 → 400提示含双方版本号ticket 05 / ADR-0001"""
client, session = client_with_db
run_id_a = _seed_run(session, "A")
sid = RunRepository(session).get(run_id_a).scenario_id
run_id_b = _seed_run(session, "B", scenario_id=sid, scenario_version=2)
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 400
detail = resp.json()["detail"]
assert "v1" in detail and "v2" in detail
def test_compare_report_same_version_ok(client_with_db):
client, session = client_with_db
run_id_a = _seed_run(session, "A", scenario_version=3)
sid = RunRepository(session).get(run_id_a).scenario_id
run_id_b = _seed_run(session, "B", scenario_id=sid, scenario_version=3)
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 200
assert resp.json()["run_a"]["scenario_version"] == 3
assert resp.json()["run_b"]["scenario_version"] == 3
def test_compare_report_run_not_found(client_with_db): def test_compare_report_run_not_found(client_with_db):
client, session = client_with_db client, session = client_with_db
run_id = _seed_run(session) run_id = _seed_run(session)

View File

@ -238,30 +238,3 @@ async def test_list_runs_includes_names_and_trigger(client, seeded_db, mock_chan
assert row["scenario_name"] == "mock-scenario" assert row["scenario_name"] == "mock-scenario"
assert row["target_name"] == "mock-target" assert row["target_name"] == "mock-target"
assert row["triggered_by"] == "ai_assistant" assert row["triggered_by"] == "ai_assistant"
# ── scenario_version snapshot (ticket 04) ────────────────────────────────
async def test_start_run_snapshots_scenario_version(client, seeded_db, mock_channel):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1",
})
assert resp.status_code == 200
assert resp.json()["scenario_version"] == 1
# 编辑考纲 → 场景升版 → 新运行快照新版本;旧运行保持 1
old_run_id = resp.json()["id"]
scenario = ScenarioRepository(seeded_db).get("s-1")
scenario.cases.append(Case(id="c2", type=CaseType.SINGLE, messages=["more"]))
ScenarioRepository(seeded_db).update(scenario)
resp2 = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "ai_assistant",
})
assert resp2.json()["scenario_version"] == 2
old = (await client.get(f"/api/runs/{old_run_id}")).json()
assert old["scenario_version"] == 1
listing = (await client.get("/api/runs")).json()
assert {row["scenario_version"] for row in listing} == {1, 2}

View File

@ -1,50 +0,0 @@
"""Exercise the scenario_version backfill migration against a real SQLite file (ticket 04)."""
from pathlib import Path
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
def test_scenario_version_backfill_migration(tmp_path: Path, monkeypatch):
from agenteval.storage import db as db_module
database_url = f"sqlite:///{tmp_path / 'migration.db'}"
monkeypatch.setattr(db_module, "DATABASE_URL", database_url)
config = Config(str(Path(__file__).resolve().parents[2] / "alembic.ini"))
legacy_engine = create_engine(database_url)
with legacy_engine.begin() as connection:
connection.execute(text("CREATE TABLE scenarios (id VARCHAR PRIMARY KEY, name VARCHAR NOT NULL)"))
connection.execute(text("CREATE TABLE eval_results (id VARCHAR PRIMARY KEY, run_id VARCHAR NOT NULL)"))
connection.execute(
text(
"CREATE TABLE eval_runs (id VARCHAR PRIMARY KEY, target_id VARCHAR NOT NULL, "
"scenario_id VARCHAR NOT NULL)"
)
)
connection.execute(text("CREATE TABLE turns (id VARCHAR PRIMARY KEY, run_id VARCHAR NOT NULL)"))
connection.execute(text("INSERT INTO scenarios (id, name) VALUES ('s1', '场景一')"))
connection.execute(
text("INSERT INTO eval_runs (id, target_id, scenario_id) VALUES ('r1', 't1', 's1')")
)
connection.execute(
text("INSERT INTO eval_runs (id, target_id, scenario_id) VALUES ('r2', 't1', 'ghost')")
)
# 先升到场景版本迁移,将 s1 手动升到 5验证回填按场景当前版本 join
command.upgrade(config, "c8e2f5a7b901")
with create_engine(database_url).begin() as connection:
connection.execute(text("UPDATE scenarios SET version = 5 WHERE id = 's1'"))
command.upgrade(config, "head")
engine = create_engine(database_url)
inspector = inspect(engine)
assert "scenario_version" in {c["name"] for c in inspector.get_columns("eval_runs")}
with engine.connect() as connection:
rows = dict(connection.execute(text("SELECT id, scenario_version FROM eval_runs")).fetchall())
assert rows["r1"] == 5 # 回填为场景当前版本
assert rows["r2"] == 1 # 孤儿运行(场景已删)回填 1

View File

@ -1,128 +0,0 @@
"""Integration tests for /api/scenarios versioning (ticket 03).
场景版本由系统维护ADR-0001仅考纲字段cases / model_bindings /
llm_config变更时递增元数据编辑不升版API 不接受外部指定版本
"""
import pytest
from cryptography.fernet import Fernet
from httpx import ASGITransport, AsyncClient
from agenteval.services.model_configs import ModelConfigService, SecretCipher
from agenteval.web.app import app
pytestmark = pytest.mark.anyio
@pytest.fixture()
def scenario_client(db_session, monkeypatch):
from agenteval.web import app as app_module
monkeypatch.setattr(app_module, "init_db", lambda: None)
from agenteval.web.deps import get_db
def _test_get_db():
try:
yield db_session
finally:
pass
app.dependency_overrides[get_db] = _test_get_db
yield db_session
app.dependency_overrides.clear()
def _client() -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
_BODY = {
"name": "版本测试场景",
"description": "初始",
"tags": ["v"],
"cases": [{"id": "c1", "type": "single", "messages": ["hi"]}],
}
async def _create(client) -> dict:
resp = await client.post("/api/scenarios", json=_BODY)
assert resp.status_code == 200, resp.text
return resp.json()
async def test_create_scenario_version_starts_at_1(scenario_client):
async with _client() as client:
created = await _create(client)
assert created["version"] == 1
async def test_editing_cases_bumps_version(scenario_client):
async with _client() as client:
created = await _create(client)
body = {**_BODY, "cases": _BODY["cases"] + [{"id": "c2", "type": "single", "messages": ["yo"]}]}
resp = await client.put(f"/api/scenarios/{created['id']}", json=body)
assert resp.status_code == 200
assert resp.json()["version"] == 2
async def test_editing_llm_config_bumps_version(scenario_client):
async with _client() as client:
created = await _create(client)
body = {**_BODY, "llm_config": {"model": "gpt-x"}}
resp = await client.put(f"/api/scenarios/{created['id']}", json=body)
assert resp.json()["version"] == 2
async def test_editing_model_bindings_bumps_version(scenario_client, db_session):
service = ModelConfigService(db_session, SecretCipher(Fernet.generate_key().decode("ascii")))
config = service.create(
name="judge", provider="openai_compatible", capability="chat",
endpoint_url="https://m.example.com/v1/chat/completions",
model_name="m", api_key="k", enabled=True, is_default=False, description="",
)
async with _client() as client:
created = await _create(client)
body = {**_BODY, "model_bindings": {"judge": config.id}}
resp = await client.put(f"/api/scenarios/{created['id']}", json=body)
assert resp.status_code == 200, resp.text
assert resp.json()["version"] == 2
async def test_metadata_edit_does_not_bump_version(scenario_client):
async with _client() as client:
created = await _create(client)
body = {**_BODY, "name": "改名了", "description": "新描述", "tags": ["x", "y"]}
resp = await client.put(f"/api/scenarios/{created['id']}", json=body)
assert resp.status_code == 200
assert resp.json()["version"] == 1
assert resp.json()["name"] == "改名了"
async def test_resaving_identical_syllabus_does_not_bump_version(scenario_client):
"""原样重存(考纲逐字节相同)不升版——序列化形态回归护栏。"""
async with _client() as client:
created = await _create(client)
resp = await client.put(f"/api/scenarios/{created['id']}", json=_BODY)
assert resp.status_code == 200
assert resp.json()["version"] == 1
async def test_external_version_is_ignored(scenario_client):
async with _client() as client:
resp = await client.post("/api/scenarios", json={**_BODY, "version": 99})
assert resp.json()["version"] == 1
created = resp.json()
body = {**_BODY, "version": 42} # 元数据未变、考纲未变 → 版本保持 1
resp = await client.put(f"/api/scenarios/{created['id']}", json=body)
assert resp.json()["version"] == 1
async def test_list_and_get_return_version(scenario_client):
async with _client() as client:
created = await _create(client)
listed = (await client.get("/api/scenarios")).json()
assert all("version" in s for s in listed)
got = (await client.get(f"/api/scenarios/{created['id']}")).json()
assert got["version"] == 1

View File

@ -332,126 +332,3 @@ async def test_dynamic_generation_failure_records_case_error(db_session):
# 被测通道不应被调用(生成阶段就失败了) # 被测通道不应被调用(生成阶段就失败了)
assert channel.send_calls == 0 assert channel.send_calls == 0
# ── expectation + explicit rules are additive (ticket 01) ────────────────
# MockChannel replies "echo: q-1", so keyword "echo" passes, "__NOPE__" fails.
from agenteval.models import EvalRuleConfig, RuleLogic # noqa: E402
def _case_with(
*,
rules: list[EvalRuleConfig] | None = None,
expectations: Expectation | None = None,
rule_logic: RuleLogic = RuleLogic.ALL,
rule_pass_threshold: float = 0.6,
) -> Case:
return Case(
id="c1", type=CaseType.SINGLE, messages=["hi"],
eval_rules=rules or [],
expectations=expectations or Expectation(),
rule_logic=rule_logic,
rule_pass_threshold=rule_pass_threshold,
)
async def test_expectation_fails_case_even_when_rules_pass(db_session):
"""期望不满足 → 用例不通过,即使显式规则全部通过。"""
scenario = Scenario(id="s1", name="s", cases=[_case_with(
rules=[EvalRuleConfig(type="keyword_match", params={"keywords": ["echo"]})],
expectations=Expectation(keywords_include=["__NOPE__"]),
)])
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.status == RunStatus.COMPLETED
assert run.summary["failed_cases"] == 1
async def test_expectation_and_rules_both_pass(db_session):
"""期望与规则都满足 → 通过且期望派生判定同构落库、reason 可辨识来源。"""
scenario = Scenario(id="s1", name="s", cases=[_case_with(
rules=[EvalRuleConfig(type="keyword_match", params={"keywords": ["echo"]})],
expectations=Expectation(keywords_include=["echo"], response_time_max_ms=99999),
)])
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.summary["passed_cases"] == 1
results = RunRepository(db_session).get_results(run.id)
# 1 显式规则 + 2 期望派生keyword + response_time
assert len(results) == 3
implicit = [r for r in results if "期望" in r.reason]
assert len(implicit) == 2
assert all(r.passed for r in results)
async def test_implicit_expectation_not_in_any_combination(db_session):
"""rule_logic=ANY 只组合显式规则:期望通过不能救活全败的显式规则组。"""
scenario = Scenario(id="s1", name="s", cases=[_case_with(
rules=[EvalRuleConfig(type="keyword_match", params={"keywords": ["__NOPE__"]})],
expectations=Expectation(keywords_include=["echo"]), # 通过
rule_logic=RuleLogic.ANY,
)])
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.summary["failed_cases"] == 1
async def test_implicit_expectation_is_hard_constraint_over_weighted(db_session):
"""rule_logic=WEIGHTED 达标但期望不满足 → 仍不通过(期望是硬约束)。"""
scenario = Scenario(id="s1", name="s", cases=[_case_with(
rules=[EvalRuleConfig(type="keyword_match", params={"keywords": ["echo"]}, weight=1.0)],
expectations=Expectation(keywords_include=["__NOPE__"]),
rule_logic=RuleLogic.WEIGHTED,
rule_pass_threshold=0.5, # 显式加权得分 1.0 ≥ 0.5
)])
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.summary["failed_cases"] == 1
async def test_pure_expectation_case_behavior_unchanged(db_session):
"""纯期望用例(无显式规则):满足通过、不满足失败,与升级前一致。"""
scenario = Scenario(id="s1", name="s", cases=[
Case(id="ok", type=CaseType.SINGLE, messages=["hi"],
expectations=Expectation(keywords_include=["echo"])),
Case(id="bad", type=CaseType.SINGLE, messages=["hi"],
expectations=Expectation(keywords_include=["__NOPE__"])),
])
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.summary["passed_cases"] == 1
assert run.summary["failed_cases"] == 1
# ── scenario_version snapshot (ticket 04) ────────────────────────────────
async def test_engine_run_snapshots_scenario_version(db_session):
"""引擎直启CLI 路径)创建的运行快照场景当前版本。"""
scenario = Scenario(
id="s-1", name="versioned", version=3,
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
)
channel = MockChannel()
engine = _build_engine(scenario, channel, session=db_session)
run = await engine.run()
assert run.scenario_version == 3
persisted = RunRepository(db_session).get(run.id)
assert persisted.scenario_version == 3

View File

@ -40,15 +40,8 @@ def _seed_run(
pass_rate: float = 1.0, pass_rate: float = 1.0,
n_cases: int = 1, n_cases: int = 1,
scenario_id: str | None = None, scenario_id: str | None = None,
scenario_version: int = 1,
connectivity_cases: int = 0,
errored_cases: int = 0,
) -> str: ) -> str:
"""Create a minimal completed run with real data in the DB and return run_id. """Create a minimal completed run with real data in the DB and return run_id."""
connectivity_cases: extra cases with replied turns but no rule results.
errored_cases: extra cases with a missing-reply turn and no rule results.
"""
target = EvalTarget( target = EvalTarget(
name="测试对象", name="测试对象",
platform=PlatformType.AI_DIGITAL_EMPLOYEE, platform=PlatformType.AI_DIGITAL_EMPLOYEE,
@ -61,11 +54,7 @@ def _seed_run(
if scenario_id is None: if scenario_id is None:
scenario = Scenario( scenario = Scenario(
name="测试场景", name="测试场景",
cases=( cases=[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)],
[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)]
+ [Case(id=f"conn{i}", type=CaseType.SINGLE, messages=["ping"]) for i in range(connectivity_cases)]
+ [Case(id=f"err{i}", type=CaseType.SINGLE, messages=["ping"]) for i in range(errored_cases)]
),
) )
scenario = ScenarioRepository(session).create(scenario) scenario = ScenarioRepository(session).create(scenario)
scenario_id = scenario.id scenario_id = scenario.id
@ -73,7 +62,6 @@ def _seed_run(
run = EvalRun( run = EvalRun(
target_id=target.id, target_id=target.id,
scenario_id=scenario_id, scenario_id=scenario_id,
scenario_version=scenario_version,
status=RunStatus.COMPLETED, status=RunStatus.COMPLETED,
) )
run = RunRepository(session).create(run) run = RunRepository(session).create(run)
@ -107,35 +95,13 @@ def _seed_run(
) )
result_repo.save_result(eval_result) result_repo.save_result(eval_result)
for i in range(connectivity_cases):
result_repo.save_turn(Turn(
run_id=run.id,
case_id=f"conn{i}",
round_index=1,
sent_message={"msgBody": {"content": "ping"}},
reply={"msgBody": {"content": "pong"}},
latency_ms=100,
))
for i in range(errored_cases):
result_repo.save_turn(Turn(
run_id=run.id,
case_id=f"err{i}",
round_index=1,
sent_message={"msgBody": {"content": "ping"}},
reply=None,
latency_ms=None,
))
all_total = total + connectivity_cases + errored_cases
all_passed = passed + connectivity_cases # 连通用例收到回复即通过(引擎口径)
run.summary = { run.summary = {
"total_cases": all_total, "total_cases": total,
"passed_cases": all_passed, "passed_cases": passed,
"failed_cases": all_total - all_passed, "failed_cases": total - passed,
"total_rules": total, "total_rules": total,
"passed_rules": passed, "passed_rules": passed,
"pass_rate": round(all_passed / all_total, 4) if all_total else 0.0, "pass_rate": round(pass_rate, 4),
} }
RunRepository(session).update(run) RunRepository(session).update(run)
return run.id return run.id
@ -244,85 +210,6 @@ def test_compare_report_different_scenarios_rejected(report_session):
generate_compare_report(run_id_a, run_id_b, report_session) generate_compare_report(run_id_a, run_id_b, report_session)
def test_compare_report_cross_version_rejected(report_session):
"""同场景不同考纲版本不可比ticket 05 / ADR-0001"""
run_id_a = _seed_run(report_session, n_cases=1)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, n_cases=1, scenario_id=sid, scenario_version=2)
with pytest.raises(ValueError, match="version"):
generate_compare_report(run_id_a, run_id_b, report_session)
def test_compare_report_same_version_allowed(report_session):
run_id_a = _seed_run(report_session, n_cases=1, scenario_version=2)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, n_cases=1, scenario_id=sid, scenario_version=2)
result = generate_compare_report(run_id_a, run_id_b, report_session)
assert result["run_a"]["scenario_version"] == 2
assert result["run_b"]["scenario_version"] == 2
# ── connectivity case annotation (ticket 02) ─────────────────────────────
def test_report_marks_connectivity_case(report_session):
run_id = _seed_run(report_session, pass_rate=0.5, n_cases=2, connectivity_cases=1)
report = generate_report(run_id, report_session)
by_id = {c["case_id"]: c for c in report["cases"]}
assert by_id["conn0"]["connectivity"] is True
assert by_id["c0"]["connectivity"] is False
assert by_id["c1"]["connectivity"] is False
s = report["summary"]
assert s["connectivity_cases"] == 1
# 判定型通过率 = 判定型通过 1 ÷ 判定型总数 2
assert s["judged_pass_rate"] == 0.5
# 总通过率口径不变含连通用例ADR-0002
assert s["pass_rate"] == round(2 / 3, 4)
def test_errored_case_not_marked_connectivity(report_session):
"""无判定结果但缺回复的用例是执行失败,不是连通用例。"""
run_id = _seed_run(report_session, n_cases=1, errored_cases=1)
report = generate_report(run_id, report_session)
by_id = {c["case_id"]: c for c in report["cases"]}
assert by_id["err0"]["connectivity"] is False
assert report["summary"]["connectivity_cases"] == 0
def test_all_connectivity_judged_pass_rate_none(report_session):
"""全为连通用例时判定型通过率为 None不除零。"""
run_id = _seed_run(report_session, n_cases=0, connectivity_cases=2)
report = generate_report(run_id, report_session)
s = report["summary"]
assert s["connectivity_cases"] == 2
assert s["judged_pass_rate"] is None
def test_compare_report_marks_connectivity(report_session):
run_id_a = _seed_run(report_session, n_cases=1, connectivity_cases=1)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, n_cases=1, connectivity_cases=1, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session)
by_id = {c["case_id"]: c for c in result["cases"]}
assert by_id["conn0"]["connectivity"] is True
assert by_id["c0"]["connectivity"] is False
# 连通用例双方均按引擎口径视为通过,不应标记 changed
assert by_id["conn0"]["run_a_passed"] is True
assert by_id["conn0"]["run_b_passed"] is True
assert by_id["conn0"]["changed"] is False
def test_markdown_report_shows_connectivity(report_session):
run_id = _seed_run(report_session, n_cases=1, connectivity_cases=1)
md = render_markdown_report(run_id, report_session)
assert "连通用例" in md
assert "判定型通过率" in md
# ── render_markdown_report ──────────────────────────────────────────────── # ── render_markdown_report ────────────────────────────────────────────────
def test_render_markdown_contains_header(report_session): def test_render_markdown_contains_header(report_session):