- _supplement_decision_logs: executing evals missing a decision log get a
platform-derived execute_session (deficit) or start_analysis (all sessions
done) entry. Audit backfill only — records observable state, does not change
agent execution. Called each scan tick after requeue+scan.
- t480 legacy cron workers (5) disabled: superseded by platform-triggered
headless agent (plan C); they kept firing every minute and failing on
Channel-required.
openclaw agent has no cron state; a bare 'agenteval-intelligent-worker'
message made the worker skill decide then 'wait for the next tick',
deadlocking (task assigned, session never created). The trigger message now
demands '立即完成当前任务，不要等待下一节拍' and, when all sessions are
done, delegates to agenteval-intelligent-analyst. Verified end-to-end on
t480: 1h-window eval went executing -> session (2 real turns) -> close ->
report -> completed, fully agent-driven, no external IM channel.
OpenClaw cron requires a channel (announce->last fail-closed); webchat is a
Control-UI feature, not an addressable channel, and platform-side static
execution would degrade the intelligent eval into a static evaluation.
Solution (plan C): the platform keeps the scan loop and, when the queue has
a pending task, invokes the headless agent:
docker exec openclaw-eval openclaw agent --agent main \
-m agenteval-intelligent-worker --json
--deliver defaults to false, so no cron delivery channel is involved. The
worker skill runs unchanged under the OpenClaw agent (LLM decisions +
evaluator/analyst skills). Verified headless invocation returns ok.
scan_and_enqueue_tasks had no scheduler: the OpenClaw Worker wakes every
minute but the platform never enqueued executing evals, so the queue was
always empty. lifespan now starts an asyncio background task that scans
executing intelligent evals every 60s (aligned with the Worker wake),
cancelled cleanly on shutdown. Verified by a new startup test (879 total).
AlertManager gains an optional openclaw_client. check_alerts records
each newly created alert and AlertManager.maybe_autoscale (called
from the async router for each alert) invokes cron_pool.scale_up(1).
scale_up itself caps at MAX_POOL_SIZE so repeated invocations are safe.
Removed the xfail guard in test_alert_autoscale_link; rewrote the test
to use task_backlog (duration_minutes=0) so a single check_alerts call
fires an alert and triggers auto-scale.
- Add list_decision_logs API endpoint
- Add DecisionProcess component with timeline, list, filter, and export
- Add decision log API calls to api.ts
- Add "决策过程" button in EvalDetail to access decision history
- Implement decision log export to JSON
- Pass TypeScript type checking
All 853 tests passing.
- Add metrics.py with pool utilization, task backlog, stuck rate, avg processing time, eval completion rate
- Add alerts.py with alert rules (pool utilization > 90%, task backlog > 50, stuck rate > 10%)
- Implement alert history and webhook notifications
- Add metrics and alerts APIs
- Add database migration for alert history table
- Add 11 unit tests for metrics, 10 unit tests for alerts, 8 integration tests
- Update migration tests to include new alert history table
All 853 tests passing.
- Add config_snapshot.py with save/list/get/compare functions
- Auto-save snapshots on eval creation and plan submission
- Implement snapshot query APIs (list, get single)
- Implement snapshot comparison API (diff two snapshots)
- Add 8 unit tests and 7 integration tests
Snapshots track config changes over time (created/plan_submitted/config_updated).
All 813 tests passing.
- Add OpenClawClient wrapping CLI commands (create/delete/list crons)
- Implement pool initialization, scale up/down, auto-scaling logic
- Implement cron state sync and stuck cron detection
- Add pool status and manual scaling APIs
- Add 13 unit tests and 5 integration tests
Pool automatically scales between 5-20 crons based on load.
All 778 tests passing.
Wrap the judge prompt and two docstrings past the 120-col convention;
record three implementation rulings in the v0.9 spec (exploration read
outlets, round-based sampling, findings carrying all ratings).
report, markdown and analysis each repeated the
summarize_exploration(repo.list_by_campaign(...)) shape; collapse it
into summarize_campaign_exploration so the aggregation has one home.
E2E on t480 showed assistant bubbles rendering {'content': '...'} because
tutu replies carry msgBody as a parsed object and the router stored
str(reply.content). Coerce to the inner text before persisting.
Exploration sessions aggregate into a single exploration summary
(session counts, goal-achievement rate, issue lists from experience
records, judge conclusions when reviewed) that feeds three exits:
the campaign report gains an exploration dimension, the v0.7 analysis
stage-two input gains the summary (stats only, never full dialogues),
and the Markdown export appends a findings appendix after analysis and
comparison. With no exploration data every output stays unchanged.
After an exploration session closes, the platform samples up to 3
conversation rounds and runs an independent judge-role review through
the v0.7 ChatClient seam, persisting quality-dimension conclusions
(attitude, professionalism, hallucination) into the session's
judge_review. The review runs as a background task: failures are
recorded without touching session state or the first-hand experience
record, and a missing model config skips silently.
Resident agents call GET /api/exploration/patrol once per cycle to see
every running production-line campaign that opted into exploration
(seed set present), the new results since the last watermark (reusing
campaign report aggregation), and the remaining exploration budget.
The watermark advances after each call so subsequent calls only report
increments; accelerated and terminal campaigns are excluded.
v0.9 ticket 02. Campaigns now carry an exploration seed set (seed
personas × seed goals — the comparability unit for exploratory
evaluation) and an optional budget override, stored as JSON columns
isomorphic to plan. Empty seeds normalize to null, marking the campaign
as opted out of exploration. resolve_budget merges per-field overrides
into platform defaults; enforcement stays server-side. The create form
gains seed lists and budget inputs (minutes → seconds), submitting null
when left empty.
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
The scheduler loop enqueues the analysis task when a realtime campaign
completes; accelerated or cancelled campaigns and a missing analysis
model skip silently. The campaign markdown export appends the analysis
appendix (overall, problems, narratives, suggestions) when a completed
analysis exists.
Add the analysis role's execution path: a two-phase orchestration
(per-scenario diagnosis gathered in parallel, then a synthesis pass)
that reads the existing campaign report aggregation plus capped failure
samples, validates the LLM's JSON against the report schema, and strips
fabricated run/scenario references before persisting. Results upsert one
row per campaign (generating/completed/failed) with the model config
snapshot; GET/POST /api/campaigns/{id}/analysis expose the state machine,
guarding non-terminal campaigns and missing analysis models with 400s.
Campaigns can pin an analysis model config instead of following the
global analysis default. Creation validates the referenced config
exists (400 otherwise); the create form offers enabled chat configs
with the global default as the fallback option.
Introduce ModelPurpose.ANALYSIS and a globally-unique is_analysis_default
marker on chat model configs so campaign analysis can resolve its model.
Service rejects disabled or non-chat configs; repo clears the previous
holder on set. Documented the analysis role in CONTEXT.md.
build_campaign_timeline flattens a campaign's child Runs into offset-sorted
per-Run entries (distinct from the report's 12-bucket aggregation), reusing a
shared _run_window_offset口径 so both views place a run identically. Exposes
GET /campaigns/{id}/timeline and the api.ts type/call. (v0.6 ticket 06)
Read paths recomputed per-case pass/connectivity independently — report
generation, the logs endpoint, and the frontend each derived it, and the
frontend's every(passed) recompute ignored the engine's authoritative
verdict. Extract resolve_case_verdicts: a single pure seam that prefers
stored case_outcomes verbatim and approximates only for legacy runs. The
logs endpoint now surfaces case_verdicts so the frontend reads instead of
recomputing.
report.py mixed DB-reading generation with string formatting: the four
render_*_report(run_id, session) functions each re-fetched via
generate_report, so the HTML/Markdown/JSON formatting was welded to storage
and could not be unit-tested from a plain dict. Extract the formatting into a
new pure report_render module whose renderers take the already-built report
dict (no session, no storage import). Migrate every caller to generate-then-
render, delete the old coupled renderers with no back-compat shim, and drop
the _aggregate_runs middle-man alias in favour of metrics.aggregate_runs.
Both the single-run path and the campaign scheduler drove long-lived
asyncio tasks through their own duplicated _tasks/_cancel_events dicts and
shutdown loops. Collapse them into one deep TaskRegistry module,
instantiated as run_registry and campaign_registry. launch() creates the
cancel event before the task (so a cancel during startup is never lost),
wires done-callback cleanup, and is idempotent per id; this makes runs.py's
hard-cancel fallback provably dead, so it is removed. App shutdown now
gracefully stops in-flight runs too, not just campaigns.
