- _supplement_decision_logs: executing evals missing a decision log get a
platform-derived execute_session (deficit) or start_analysis (all sessions
done) entry. Audit backfill only — records observable state, does not change
agent execution. Called each scan tick after requeue+scan.
- t480 legacy cron workers (5) disabled: superseded by platform-triggered
headless agent (plan C); they kept firing every minute and failing on
Channel-required.
openclaw agent has no cron state; a bare 'agenteval-intelligent-worker'
message made the worker skill decide then 'wait for the next tick',
deadlocking (task assigned, session never created). The trigger message now
demands '立即完成当前任务，不要等待下一节拍' and, when all sessions are
done, delegates to agenteval-intelligent-analyst. Verified end-to-end on
t480: 1h-window eval went executing -> session (2 real turns) -> close ->
report -> completed, fully agent-driven, no external IM channel.
OpenClaw cron requires a channel (announce->last fail-closed); webchat is a
Control-UI feature, not an addressable channel, and platform-side static
execution would degrade the intelligent eval into a static evaluation.
Solution (plan C): the platform keeps the scan loop and, when the queue has
a pending task, invokes the headless agent:
docker exec openclaw-eval openclaw agent --agent main \
-m agenteval-intelligent-worker --json
--deliver defaults to false, so no cron delivery channel is involved. The
worker skill runs unchanged under the OpenClaw agent (LLM decisions +
evaluator/analyst skills). Verified headless invocation returns ok.
scan_and_enqueue_tasks had no scheduler: the OpenClaw Worker wakes every
minute but the platform never enqueued executing evals, so the queue was
always empty. lifespan now starts an asyncio background task that scans
executing intelligent evals every 60s (aligned with the Worker wake),
cancelled cleanly on shutdown. Verified by a new startup test (879 total).
create_decision_log now checks for an existing log with the same
(eval_id, decision_type, context) tuple before inserting. If found, it
returns the existing row's dict instead of appending a duplicate. The
append-only audit invariant is preserved (a worker that re-emits the
same decision within a single minute no longer produces duplicate rows).
Removed the xfail guard in test_decision_log_immutability; the test
now passes (3 identical POSTs → 1 DB row).
AlertManager gains an optional openclaw_client. check_alerts records
each newly created alert and AlertManager.maybe_autoscale (called
from the async router for each alert) invokes cron_pool.scale_up(1).
scale_up itself caps at MAX_POOL_SIZE so repeated invocations are safe.
Removed the xfail guard in test_alert_autoscale_link; rewrote the test
to use task_backlog (duration_minutes=0) so a single check_alerts call
fires an alert and triggers auto-scale.
Replace read-check-write in task_queue.assign_task with UPDATE...WHERE
status='pending' and decide on rowcount so two concurrent workers
cannot both claim the same task. Also harden complete_task with the
same CAS pattern (status='assigned') so a worker + stuck-handler
double-complete leaves the DB in one state.
The xfail guard in test_worker_task_resilience now passes (4/4).
T2 frontend CronPoolMonitor: polls every 5s, unmount clears interval,
plus it.fails guard for missing visibilitychange listener (S6).
T3 router ORM contract: heartbeat updates fields + 404, decision-logs
POST persists + 404, GET lists inserted (guards S2 — must continue to
pass after router handlers move into a service in P3).
T8 decision-log immutability: append-only on context change passes;
dedupe of identical (decision_type, context) is xfail (real gap).
T7 alert→auto-scale: task_backlog alert recorded on first call (passes);
check_alerts never invokes auto_scale is xfail (real gap, §6.4).
All real gaps are logged in .scratch/v111-architecture-scan.md §6.
- Add metrics.py with pool utilization, task backlog, stuck rate, avg processing time, eval completion rate
- Add alerts.py with alert rules (pool utilization > 90%, task backlog > 50, stuck rate > 10%)
- Implement alert history and webhook notifications
- Add metrics and alerts APIs
- Add database migration for alert history table
- Add 11 unit tests for metrics, 10 unit tests for alerts, 8 integration tests
- Update migration tests to include new alert history table
All 853 tests passing.
- Add config_snapshot.py with save/list/get/compare functions
- Auto-save snapshots on eval creation and plan submission
- Implement snapshot query APIs (list, get single)
- Implement snapshot comparison API (diff two snapshots)
- Add 8 unit tests and 7 integration tests
Snapshots track config changes over time (created/plan_submitted/config_updated).
All 813 tests passing.
- Add OpenClawClient wrapping CLI commands (create/delete/list crons)
- Implement pool initialization, scale up/down, auto-scaling logic
- Implement cron state sync and stuck cron detection
- Add pool status and manual scaling APIs
- Add 13 unit tests and 5 integration tests
Pool automatically scales between 5-20 crons based on load.
All 778 tests passing.
E2E on t480 showed assistant bubbles rendering {'content': '...'} because
tutu replies carry msgBody as a parsed object and the router stored
str(reply.content). Coerce to the inner text before persisting.
Exploration sessions aggregate into a single exploration summary
(session counts, goal-achievement rate, issue lists from experience
records, judge conclusions when reviewed) that feeds three exits:
the campaign report gains an exploration dimension, the v0.7 analysis
stage-two input gains the summary (stats only, never full dialogues),
and the Markdown export appends a findings appendix after analysis and
comparison. With no exploration data every output stays unchanged.
After an exploration session closes, the platform samples up to 3
conversation rounds and runs an independent judge-role review through
the v0.7 ChatClient seam, persisting quality-dimension conclusions
(attitude, professionalism, hallucination) into the session's
judge_review. The review runs as a background task: failures are
recorded without touching session state or the first-hand experience
record, and a missing model config skips silently.
Resident agents call GET /api/exploration/patrol once per cycle to see
every running production-line campaign that opted into exploration
(seed set present), the new results since the last watermark (reusing
campaign report aggregation), and the remaining exploration budget.
The watermark advances after each call so subsequent calls only report
increments; accelerated and terminal campaigns are excluded.
v0.9 ticket 02. Campaigns now carry an exploration seed set (seed
personas × seed goals — the comparability unit for exploratory
evaluation) and an optional budget override, stored as JSON columns
isomorphic to plan. Empty seeds normalize to null, marking the campaign
as opted out of exploration. resolve_budget merges per-field overrides
into platform defaults; enforcement stays server-side. The create form
gains seed lists and budget inputs (minutes → seconds), submitting null
when left empty.
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
The scheduler loop enqueues the analysis task when a realtime campaign
completes; accelerated or cancelled campaigns and a missing analysis
model skip silently. The campaign markdown export appends the analysis
appendix (overall, problems, narratives, suggestions) when a completed
analysis exists.
Add the analysis role's execution path: a two-phase orchestration
(per-scenario diagnosis gathered in parallel, then a synthesis pass)
that reads the existing campaign report aggregation plus capped failure
samples, validates the LLM's JSON against the report schema, and strips
fabricated run/scenario references before persisting. Results upsert one
row per campaign (generating/completed/failed) with the model config
snapshot; GET/POST /api/campaigns/{id}/analysis expose the state machine,
guarding non-terminal campaigns and missing analysis models with 400s.
Campaigns can pin an analysis model config instead of following the
global analysis default. Creation validates the referenced config
exists (400 otherwise); the create form offers enabled chat configs
with the global default as the fallback option.
Introduce ModelPurpose.ANALYSIS and a globally-unique is_analysis_default
marker on chat model configs so campaign analysis can resolve its model.
Service rejects disabled or non-chat configs; repo clears the previous
holder on set. Documented the analysis role in CONTEXT.md.
build_campaign_timeline flattens a campaign's child Runs into offset-sorted
per-Run entries (distinct from the report's 12-bucket aggregation), reusing a
shared _run_window_offset口径 so both views place a run identically. Exposes
GET /campaigns/{id}/timeline and the api.ts type/call. (v0.6 ticket 06)
Both the single-run path and the campaign scheduler drove long-lived
asyncio tasks through their own duplicated _tasks/_cancel_events dicts and
shutdown loops. Collapse them into one deep TaskRegistry module,
instantiated as run_registry and campaign_registry. launch() creates the
cancel event before the task (so a cancel during startup is never lost),
wires done-callback cleanup, and is idempotent per id; this makes runs.py's
hard-cancel fallback provably dead, so it is removed. App shutdown now
gracefully stops in-flight runs too, not just campaigns.
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
Embed compact progress (completed/planned total + overall pass_rate,
reusing the report's aggregation) into GET /campaigns so the list drops
its N+1 report fetch. Poll list and open report drawer every 5s while the
tab is active and a campaign is still running. Show scenario version and
trigger source tags in the child-run drill-down.
Add generate_campaign_report: a pure aggregator over a campaign's child Runs
producing a time-trend axis (Runs bucketed by service-window position) and a
capability-summary axis (grouped by scenario), each carrying pass_rate /
availability / latency. pass_rate keeps the single-Run case-level meaning and
counts execution failures as 0.0 (ADR-0002); time_scale only places Runs into
window-time buckets and never alters any figure. Engine summary now records
avg_latency_ms to feed the latency axis.
Expose GET /api/campaigns/{id}/report (structured) and .../report/markdown
(reusing the existing Markdown export path). Adds "可用性/Availability" to the
domain glossary.
Add a thin async loop (run_campaign_loop) that ticks on real wall-clock time,
maps elapsed×time_scale to a window offset via the pure decide_schedule, spawns
due child Runs, and marks the campaign COMPLETED at window end. All authority
lives in the DB (started_at, spawned_indices, status), so the app lifespan can
resume every RUNNING campaign on startup without double-spawning and stop all
loops gracefully on shutdown. A failing plan entry is skipped and recorded
rather than wedging the campaign.
Creating a campaign now starts its loop; POST /api/campaigns/{id}/cancel stops
further spawning (completed child Runs are kept); GET /api/campaigns/{id}
reports live progress (window offset, spawned/completed Run counts).
