After an exploration session closes, the platform samples up to 3
conversation rounds and runs an independent judge-role review through
the v0.7 ChatClient seam, persisting quality-dimension conclusions
(attitude, professionalism, hallucination) into the session's
judge_review. The review runs as a background task: failures are
recorded without touching session state or the first-hand experience
record, and a missing model config skips silently.
Resident agents call GET /api/exploration/patrol once per cycle to see
every running production-line campaign that opted into exploration
(seed set present), the new results since the last watermark (reusing
campaign report aggregation), and the remaining exploration budget.
The watermark advances after each call so subsequent calls only report
increments; accelerated and terminal campaigns are excluded.
v0.9 ticket 02. Campaigns now carry an exploration seed set (seed
personas × seed goals — the comparability unit for exploratory
evaluation) and an optional budget override, stored as JSON columns
isomorphic to plan. Empty seeds normalize to null, marking the campaign
as opted out of exploration. resolve_budget merges per-field overrides
into platform defaults; enforcement stays server-side. The create form
gains seed lists and budget inputs (minutes → seconds), submitting null
when left empty.
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
Record the Campaign milestone (v0.6: durable scheduler, dual-axis
report, campaigns page) and the campaign-intelligence milestone (v0.7:
two-phase analysis agent, analysis model default/override, drawer
section, auto-trigger and markdown export). Bump the single-source
version to 0.7.0.
The scheduler loop enqueues the analysis task when a realtime campaign
completes; accelerated or cancelled campaigns and a missing analysis
model skip silently. The campaign markdown export appends the analysis
appendix (overall, problems, narratives, suggestions) when a completed
analysis exists.
Render the campaign analysis in the report drawer: status row for
generating (5s polling), failed (error + retry) and empty states, then
the structured result — overall callout, problem cards with severity
tags and evidence chips linking to run reports, per-scenario narratives
and priority-sorted suggestions. Generate buttons are terminal-only with
guidance when no analysis model is configured.
Add the analysis role's execution path: a two-phase orchestration
(per-scenario diagnosis gathered in parallel, then a synthesis pass)
that reads the existing campaign report aggregation plus capped failure
samples, validates the LLM's JSON against the report schema, and strips
fabricated run/scenario references before persisting. Results upsert one
row per campaign (generating/completed/failed) with the model config
snapshot; GET/POST /api/campaigns/{id}/analysis expose the state machine,
guarding non-terminal campaigns and missing analysis models with 400s.
Campaigns can pin an analysis model config instead of following the
global analysis default. Creation validates the referenced config
exists (400 otherwise); the create form offers enabled chat configs
with the global default as the fallback option.
Introduce ModelPurpose.ANALYSIS and a globally-unique is_analysis_default
marker on chat model configs so campaign analysis can resolve its model.
Service rejects disabled or non-chat configs; repo clears the previous
holder on set. Documented the analysis role in CONTEXT.md.
Report drawer widens to 1040px with a four-card metric row (rate-graded
colours), the embedded scenario-lane process timeline with now-line,
a dual-series pass-rate/availability trend next to a horizontal
capability ranking bar chart, and a filterable/sortable sub-run table
with a latency column. Chart configs move to the charts v2 scale/axis
API — the old yAxis key was dead v1 config.
Plan preview markers are plain colour blocks (names live in the
tooltip and legend); blocks that would overlap spread onto staggered
rows. Plan start time is limited to the window length via InputNumber
max plus a validator, re-checked immediately when the window shrinks.
Create surface moves from a 640px modal to a 920px two-column drawer:
basic info + time/speed on the left, plan preview and a grid-aligned
plan editor (headers, searchable scenario selects, scrollable entries)
on the right, so many-entry plans stay editable. The expanded-row
process timeline switches from a single crowded axis to per-scenario
lanes with a now-line for active campaigns.
Each campaign row expands to a per-Run process timeline: on expand it fetches
GET /campaigns/{id}/timeline and places each child Run on the shared
WindowTimeline by accelerated window offset, coloured by run status, with
scenario/pass-rate/latency tooltips and click-through to the run report.
Running campaigns refresh on the existing 5s poll; terminal ones fetch once.
WindowTimeline gains a colorMap prop for semantic status colours. (v0.6 ticket 10)
Above the plan editor, adapt each plan entry (offset, scenario, count) into
WindowTimeline markers coloured per scenario with a count badge, re-rendering
live as the form changes via Form.useWatch. Empty plan shows a placeholder.
Purely form-local, no backend change. (v0.6 ticket 09)
A domain-agnostic horizontal timeline: an axis over the service-cycle window
with offset-positioned markers, stable per-colorKey colours, optional badges,
hover tooltips, click callbacks, and an optional legend. Shared base for the
plan preview (09) and process timeline (10). (v0.6 ticket 08)
Users now pick a window length + "how long it should actually take" and the
form derives time_scale (window ÷ target), showing 倍速 ×N and 加速后耗时
read-only; a 实时 switch pins real wall-clock (×1). Validation blocks a target
longer than the window. List/detail display accelerated duration instead of raw
×N. POST /campaigns contract unchanged. (v0.6 ticket 07)
build_campaign_timeline flattens a campaign's child Runs into offset-sorted
per-Run entries (distinct from the report's 12-bucket aggregation), reusing a
shared _run_window_offset口径 so both views place a run identically. Exposes
GET /campaigns/{id}/timeline and the api.ts type/call. (v0.6 ticket 06)
Campaign.summary was a bare Optional[dict] while RunSummary is a typed VO —
scheduler state (spawned_indices/errors) flowed untyped through
set_/get_summary. Introduce CampaignSummary + SchedulerState (extra=allow,
validate_assignment), mirroring RunSummary; campaign_runner reads/writes the
VO. Also converge the ~10 repeated JSON column get/set pairs onto
_json_dumps/_json_loads helpers, unifying ensure_ascii=False and fixing the
set_modalities ensure_ascii=True trap.
Three near-identical Turn(...)+save_turn blocks (send-fail / poll-except /
happy path) collapse into one _persist_turn helper differing only by the
optional fields set. The期望→隐式规则 translation becomes a pure,
unit-testable derive_implicit_rules seam (CONTEXT: 期望与规则叠加生效),
and the rule_type→ModelPurpose map is hoisted to a module constant.
Single-run summary口径 (pass_rate / judged_pass_rate / avg_latency /
connectivity split) was inlined in run(), reachable only by driving a
whole async run, and report.py recomputed judged_pass_rate independently.
Extract build_run_summary — a pure function parallel to aggregate_runs
(cross-run) and combine_case_outcome (case-level). run() now collects
material and delegates; judged_pass_rate is stored in RunSummary so the
report reads it instead of recomputing.
Read paths recomputed per-case pass/connectivity independently — report
generation, the logs endpoint, and the frontend each derived it, and the
frontend's every(passed) recompute ignored the engine's authoritative
verdict. Extract resolve_case_verdicts: a single pure seam that prefers
stored case_outcomes verbatim and approximates only for legacy runs. The
logs endpoint now surfaces case_verdicts so the frontend reads instead of
recomputing.
Seven pages repeated the same load-on-mount + loading + try/finally +
reload-button skeleton, each re-implementing tab-active refresh, silent
polling, and (in two places) a hand-rolled requestId race guard. Extract two
composable hooks: useResource(fetcher, {tabPath, deps}) owning data/loading/
reload with a built-in race guard and auto tab-active refresh, and
usePolling(fn, ms, enabled) replacing the hand-written setInterval effects.
Migrate all seven pages onto them; Targets/Scenarios/ModelConfigs also gain a
uniform tab-active refresh they previously lacked. Verified via tsc --noEmit
and npm run build (no frontend test runner exists).
report.py mixed DB-reading generation with string formatting: the four
render_*_report(run_id, session) functions each re-fetched via
generate_report, so the HTML/Markdown/JSON formatting was welded to storage
and could not be unit-tested from a plain dict. Extract the formatting into a
new pure report_render module whose renderers take the already-built report
dict (no session, no storage import). Migrate every caller to generate-then-
render, delete the old coupled renderers with no back-compat shim, and drop
the _aggregate_runs middle-man alias in favour of metrics.aggregate_runs.
Both the single-run path and the campaign scheduler drove long-lived
asyncio tasks through their own duplicated _tasks/_cancel_events dicts and
shutdown loops. Collapse them into one deep TaskRegistry module,
instantiated as run_registry and campaign_registry. launch() creates the
cancel event before the task (so a cancel during startup is never lost),
wires done-callback cleanup, and is idempotent per id; this makes runs.py's
hard-cancel fallback provably dead, so it is removed. App shutdown now
gracefully stops in-flight runs too, not just campaigns.
Extend the pure scheduler with elapsed_seconds (clock injected), decide_tick
(offset + due + lifecycle action) and resolve_finalize (cancel-race guard),
so the durable loop stops hand-coding elapsed/finished/status checks and only
does I/O. Deletes the runner's private _elapsed_seconds and converges
current_window_offset onto the one pure elapsed computation. The clock-skew
tolerance and cancel-race guard are now unit-testable at the seam.
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
Embed compact progress (completed/planned total + overall pass_rate,
reusing the report's aggregation) into GET /campaigns so the list drops
its N+1 report fetch. Poll list and open report drawer every 5s while the
tab is active and a campaign is still running. Show scenario version and
trigger source tags in the child-run drill-down.
Register a keep-alive "评估活动" tab that creates campaigns (target,
window, time_scale, static plan), lists them with live progress and
pass-rate, and opens a report drawer with a time-trend line, capability
summary, and drill-down into child Runs.
Add generate_campaign_report: a pure aggregator over a campaign's child Runs
producing a time-trend axis (Runs bucketed by service-window position) and a
capability-summary axis (grouped by scenario), each carrying pass_rate /
availability / latency. pass_rate keeps the single-Run case-level meaning and
counts execution failures as 0.0 (ADR-0002); time_scale only places Runs into
window-time buckets and never alters any figure. Engine summary now records
avg_latency_ms to feed the latency axis.
Expose GET /api/campaigns/{id}/report (structured) and .../report/markdown
(reusing the existing Markdown export path). Adds "可用性/Availability" to the
domain glossary.
Add a thin async loop (run_campaign_loop) that ticks on real wall-clock time,
maps elapsed×time_scale to a window offset via the pure decide_schedule, spawns
due child Runs, and marks the campaign COMPLETED at window end. All authority
lives in the DB (started_at, spawned_indices, status), so the app lifespan can
resume every RUNNING campaign on startup without double-spawning and stop all
loops gracefully on shutdown. A failing plan entry is skipped and recorded
rather than wedging the campaign.
Creating a campaign now starts its loop; POST /api/campaigns/{id}/cancel stops
further spawning (completed child Runs are kept); GET /api/campaigns/{id}
reports live progress (window offset, spawned/completed Run counts).
Add the pure scheduling seam (campaign_scheduler.decide_schedule) that, given a
static plan and window-clock offset, decides which plan entries are due and
whether the window ended — mirroring judgement.combine_case_outcome, with
time_scale confined to the clock mapping so it never touches judgement/report.
The campaign_runner shell maps injected elapsed time to a window offset, spawns
due child Runs through the existing EvalEngine.run(existing_run=...) path with
campaign_id + RunTrigger.CAMPAIGN, and persists spawned-entry indices per entry
for idempotent, restart-recoverable progress. No auto loop yet (ticket 03).
