12481cd1b8
## 核心变更
### 规则层全面异步化（DEBT-1）
- EvalRule.evaluate() 签名改为 async def，全量同步改造（无兼容层）
- LlmScoreRule._call_llm: requests.post → httpx.AsyncClient，彻底消除事件循环阻塞
- engine._save_rule_results: rule.evaluate() → await rule.evaluate()
### 工具函数去重（DEBT-2）
- 新建 agenteval/utils/llm.py，统一三个函数：
- extract_reply_text (原 5 处重复)
- extract_content_from_llm_response (原 2 处重复)
- parse_json_from_llm_text (统一 LLM 输出 JSON 解析)
- engine.py / llm_score.py / runs.py / report.py 全部切换到 utils.llm
### HTTP 通用通道（S1-3）
- 新建 channels/http.py (HttpChannel)
- 配置化 send_url / reply_url 模板 ({message}, {msg_id} 占位)
- dot-path 提取 msg_id 和 reply_text
- 可选 reply_ready_path 就绪标志
- 长连接 AsyncClient 复用
- ChannelFactory 注册 ChannelType.HTTP → HttpChannel
### 测试
- 新增 tests/unit/test_http_channel_and_rules.py (19 个测试)
- _get_path / health_check / send / poll_reply / 超时 / 就绪标志 / async 规则评估
- 测试总数：24 → 43，全部通过
Co-Authored-By: Claude <noreply@anthropic.com>
39 lines
1.4 KiB
Python
39 lines
1.4 KiB
Python
"""Keyword matching evaluation rule."""
|
|
|
|
from agenteval.evaluation.rules.base import EvalRule, RuleResult, register_rule
|
|
from agenteval.models import Case, Turn
|
|
from agenteval.utils.llm import extract_reply_text
|
|
|
|
|
|
@register_rule
|
|
class KeywordMatchRule(EvalRule):
|
|
"""Check whether the reply contains required keywords and excludes forbidden ones."""
|
|
|
|
name = "keyword_match"
|
|
|
|
async def evaluate(self, case: Case, dialog: list[Turn]) -> RuleResult:
|
|
if not dialog:
|
|
return RuleResult(passed=False, reason="无回复记录")
|
|
|
|
last_turn = dialog[-1]
|
|
text = extract_reply_text(last_turn.reply).lower()
|
|
|
|
params = self.params
|
|
include = [k.lower() for k in params.get("keywords", [])]
|
|
exclude = [k.lower() for k in params.get("exclude_keywords", [])]
|
|
|
|
missing = [k for k in include if k not in text]
|
|
found_excluded = [k for k in exclude if k in text]
|
|
|
|
if missing or found_excluded:
|
|
reasons = []
|
|
if missing:
|
|
reasons.append(f"缺少关键词: {missing}")
|
|
if found_excluded:
|
|
reasons.append(f"包含禁用词: {found_excluded}")
|
|
return RuleResult(passed=False, reason="; ".join(reasons))
|
|
|
|
match_count = sum(1 for k in include if k in text)
|
|
score = 1.0 if not include else match_count / len(include)
|
|
return RuleResult(passed=True, score=score, reason="关键词匹配通过")
|