12481cd1b8
## 核心变更
### 规则层全面异步化（DEBT-1）
- EvalRule.evaluate() 签名改为 async def，全量同步改造（无兼容层）
- LlmScoreRule._call_llm: requests.post → httpx.AsyncClient，彻底消除事件循环阻塞
- engine._save_rule_results: rule.evaluate() → await rule.evaluate()
### 工具函数去重（DEBT-2）
- 新建 agenteval/utils/llm.py，统一三个函数：
- extract_reply_text (原 5 处重复)
- extract_content_from_llm_response (原 2 处重复)
- parse_json_from_llm_text (统一 LLM 输出 JSON 解析)
- engine.py / llm_score.py / runs.py / report.py 全部切换到 utils.llm
### HTTP 通用通道（S1-3）
- 新建 channels/http.py (HttpChannel)
- 配置化 send_url / reply_url 模板 ({message}, {msg_id} 占位)
- dot-path 提取 msg_id 和 reply_text
- 可选 reply_ready_path 就绪标志
- 长连接 AsyncClient 复用
- ChannelFactory 注册 ChannelType.HTTP → HttpChannel
### 测试
- 新增 tests/unit/test_http_channel_and_rules.py (19 个测试)
- _get_path / health_check / send / poll_reply / 超时 / 就绪标志 / async 规则评估
- 测试总数：24 → 43，全部通过
Co-Authored-By: Claude <noreply@anthropic.com>
50 lines
1.2 KiB
Python
50 lines
1.2 KiB
Python
"""Base class for evaluation rules."""
|
|
|
|
from abc import ABC, abstractmethod
|
|
from dataclasses import dataclass
|
|
from typing import Any, Optional
|
|
|
|
from agenteval.models import Case, Turn
|
|
|
|
|
|
@dataclass
|
|
class RuleResult:
|
|
"""Result of applying an evaluation rule."""
|
|
|
|
passed: bool
|
|
score: Optional[float] = None
|
|
reason: str = ""
|
|
|
|
|
|
class EvalRule(ABC):
|
|
"""Abstract evaluation rule. All implementations must be async."""
|
|
|
|
name: str = ""
|
|
|
|
def __init__(self, params: dict[str, Any]):
|
|
self.params = params
|
|
|
|
@abstractmethod
|
|
async def evaluate(self, case: Case, dialog: list[Turn]) -> RuleResult:
|
|
"""Evaluate the dialog against the case expectation."""
|
|
...
|
|
|
|
|
|
_RULE_REGISTRY: dict[str, type[EvalRule]] = {}
|
|
|
|
|
|
def register_rule(rule_cls: type[EvalRule]) -> type[EvalRule]:
|
|
_RULE_REGISTRY[rule_cls.name] = rule_cls
|
|
return rule_cls
|
|
|
|
|
|
def get_rule(rule_type: str, params: dict[str, Any]) -> EvalRule:
|
|
"""Instantiate a rule by type name."""
|
|
if rule_type not in _RULE_REGISTRY:
|
|
raise ValueError(f"unknown rule type: {rule_type}. Available: {list(_RULE_REGISTRY.keys())}")
|
|
return _RULE_REGISTRY[rule_type](params)
|
|
|
|
|
|
def list_rule_types() -> list[str]:
|
|
return list(_RULE_REGISTRY.keys())
|