v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
Record the Campaign milestone (v0.6: durable scheduler, dual-axis
report, campaigns page) and the campaign-intelligence milestone (v0.7:
two-phase analysis agent, analysis model default/override, drawer
section, auto-trigger and markdown export). Bump the single-source
version to 0.7.0.
The scheduler loop enqueues the analysis task when a realtime campaign
completes; accelerated or cancelled campaigns and a missing analysis
model skip silently. The campaign markdown export appends the analysis
appendix (overall, problems, narratives, suggestions) when a completed
analysis exists.
Render the campaign analysis in the report drawer: status row for
generating (5s polling), failed (error + retry) and empty states, then
the structured result — overall callout, problem cards with severity
tags and evidence chips linking to run reports, per-scenario narratives
and priority-sorted suggestions. Generate buttons are terminal-only with
guidance when no analysis model is configured.
Add the analysis role's execution path: a two-phase orchestration
(per-scenario diagnosis gathered in parallel, then a synthesis pass)
that reads the existing campaign report aggregation plus capped failure
samples, validates the LLM's JSON against the report schema, and strips
fabricated run/scenario references before persisting. Results upsert one
row per campaign (generating/completed/failed) with the model config
snapshot; GET/POST /api/campaigns/{id}/analysis expose the state machine,
guarding non-terminal campaigns and missing analysis models with 400s.
Campaigns can pin an analysis model config instead of following the
global analysis default. Creation validates the referenced config
exists (400 otherwise); the create form offers enabled chat configs
with the global default as the fallback option.
Introduce ModelPurpose.ANALYSIS and a globally-unique is_analysis_default
marker on chat model configs so campaign analysis can resolve its model.
Service rejects disabled or non-chat configs; repo clears the previous
holder on set. Documented the analysis role in CONTEXT.md.
Report drawer widens to 1040px with a four-card metric row (rate-graded
colours), the embedded scenario-lane process timeline with now-line,
a dual-series pass-rate/availability trend next to a horizontal
capability ranking bar chart, and a filterable/sortable sub-run table
with a latency column. Chart configs move to the charts v2 scale/axis
API — the old yAxis key was dead v1 config.
Plan preview markers are plain colour blocks (names live in the
tooltip and legend); blocks that would overlap spread onto staggered
rows. Plan start time is limited to the window length via InputNumber
max plus a validator, re-checked immediately when the window shrinks.
Create surface moves from a 640px modal to a 920px two-column drawer:
basic info + time/speed on the left, plan preview and a grid-aligned
plan editor (headers, searchable scenario selects, scrollable entries)
on the right, so many-entry plans stay editable. The expanded-row
process timeline switches from a single crowded axis to per-scenario
lanes with a now-line for active campaigns.
Each campaign row expands to a per-Run process timeline: on expand it fetches
GET /campaigns/{id}/timeline and places each child Run on the shared
WindowTimeline by accelerated window offset, coloured by run status, with
scenario/pass-rate/latency tooltips and click-through to the run report.
Running campaigns refresh on the existing 5s poll; terminal ones fetch once.
WindowTimeline gains a colorMap prop for semantic status colours. (v0.6 ticket 10)
Above the plan editor, adapt each plan entry (offset, scenario, count) into
WindowTimeline markers coloured per scenario with a count badge, re-rendering
live as the form changes via Form.useWatch. Empty plan shows a placeholder.
Purely form-local, no backend change. (v0.6 ticket 09)
A domain-agnostic horizontal timeline: an axis over the service-cycle window
with offset-positioned markers, stable per-colorKey colours, optional badges,
hover tooltips, click callbacks, and an optional legend. Shared base for the
plan preview (09) and process timeline (10). (v0.6 ticket 08)
Users now pick a window length + "how long it should actually take" and the
form derives time_scale (window ÷ target), showing 倍速 ×N and 加速后耗时
read-only; a 实时 switch pins real wall-clock (×1). Validation blocks a target
longer than the window. List/detail display accelerated duration instead of raw
×N. POST /campaigns contract unchanged. (v0.6 ticket 07)
build_campaign_timeline flattens a campaign's child Runs into offset-sorted
per-Run entries (distinct from the report's 12-bucket aggregation), reusing a
shared _run_window_offset口径 so both views place a run identically. Exposes
GET /campaigns/{id}/timeline and the api.ts type/call. (v0.6 ticket 06)
Campaign.summary was a bare Optional[dict] while RunSummary is a typed VO —
scheduler state (spawned_indices/errors) flowed untyped through
set_/get_summary. Introduce CampaignSummary + SchedulerState (extra=allow,
validate_assignment), mirroring RunSummary; campaign_runner reads/writes the
VO. Also converge the ~10 repeated JSON column get/set pairs onto
_json_dumps/_json_loads helpers, unifying ensure_ascii=False and fixing the
set_modalities ensure_ascii=True trap.
Three near-identical Turn(...)+save_turn blocks (send-fail / poll-except /
happy path) collapse into one _persist_turn helper differing only by the
optional fields set. The期望→隐式规则 translation becomes a pure,
unit-testable derive_implicit_rules seam (CONTEXT: 期望与规则叠加生效),
and the rule_type→ModelPurpose map is hoisted to a module constant.
Single-run summary口径 (pass_rate / judged_pass_rate / avg_latency /
connectivity split) was inlined in run(), reachable only by driving a
whole async run, and report.py recomputed judged_pass_rate independently.
Extract build_run_summary — a pure function parallel to aggregate_runs
(cross-run) and combine_case_outcome (case-level). run() now collects
material and delegates; judged_pass_rate is stored in RunSummary so the
report reads it instead of recomputing.
Read paths recomputed per-case pass/connectivity independently — report
generation, the logs endpoint, and the frontend each derived it, and the
frontend's every(passed) recompute ignored the engine's authoritative
verdict. Extract resolve_case_verdicts: a single pure seam that prefers
stored case_outcomes verbatim and approximates only for legacy runs. The
logs endpoint now surfaces case_verdicts so the frontend reads instead of
recomputing.
Seven pages repeated the same load-on-mount + loading + try/finally +
reload-button skeleton, each re-implementing tab-active refresh, silent
polling, and (in two places) a hand-rolled requestId race guard. Extract two
composable hooks: useResource(fetcher, {tabPath, deps}) owning data/loading/
reload with a built-in race guard and auto tab-active refresh, and
usePolling(fn, ms, enabled) replacing the hand-written setInterval effects.
Migrate all seven pages onto them; Targets/Scenarios/ModelConfigs also gain a
uniform tab-active refresh they previously lacked. Verified via tsc --noEmit
and npm run build (no frontend test runner exists).
report.py mixed DB-reading generation with string formatting: the four
render_*_report(run_id, session) functions each re-fetched via
generate_report, so the HTML/Markdown/JSON formatting was welded to storage
and could not be unit-tested from a plain dict. Extract the formatting into a
new pure report_render module whose renderers take the already-built report
dict (no session, no storage import). Migrate every caller to generate-then-
render, delete the old coupled renderers with no back-compat shim, and drop
the _aggregate_runs middle-man alias in favour of metrics.aggregate_runs.
Both the single-run path and the campaign scheduler drove long-lived
asyncio tasks through their own duplicated _tasks/_cancel_events dicts and
shutdown loops. Collapse them into one deep TaskRegistry module,
instantiated as run_registry and campaign_registry. launch() creates the
cancel event before the task (so a cancel during startup is never lost),
wires done-callback cleanup, and is idempotent per id; this makes runs.py's
hard-cancel fallback provably dead, so it is removed. App shutdown now
gracefully stops in-flight runs too, not just campaigns.
Extend the pure scheduler with elapsed_seconds (clock injected), decide_tick
(offset + due + lifecycle action) and resolve_finalize (cancel-race guard),
so the durable loop stops hand-coding elapsed/finished/status checks and only
does I/O. Deletes the runner's private _elapsed_seconds and converges
current_window_offset onto the one pure elapsed computation. The clock-skew
tolerance and cancel-race guard are now unit-testable at the seam.
Give EvalRun.summary a typed RunSummary value (unified RunError, lenient
legacy parsing) so readers stop reaching into a schemaless dict, and route
every cross-run rollup — dashboard, scenario ranking, trend, campaign
report — through one aggregate_runs seam. Fixes the divergence where
stats averaged pass_rate over completed-only runs while the campaign
report counted faults as 0.0. Cross-run rule (ADR-0004): genuine faults
count 0.0, user-cancelled runs are excluded from both denominators.
Embed compact progress (completed/planned total + overall pass_rate,
reusing the report's aggregation) into GET /campaigns so the list drops
its N+1 report fetch. Poll list and open report drawer every 5s while the
tab is active and a campaign is still running. Show scenario version and
trigger source tags in the child-run drill-down.
Register a keep-alive "评估活动" tab that creates campaigns (target,
window, time_scale, static plan), lists them with live progress and
pass-rate, and opens a report drawer with a time-trend line, capability
summary, and drill-down into child Runs.
Add generate_campaign_report: a pure aggregator over a campaign's child Runs
producing a time-trend axis (Runs bucketed by service-window position) and a
capability-summary axis (grouped by scenario), each carrying pass_rate /
availability / latency. pass_rate keeps the single-Run case-level meaning and
counts execution failures as 0.0 (ADR-0002); time_scale only places Runs into
window-time buckets and never alters any figure. Engine summary now records
avg_latency_ms to feed the latency axis.
Expose GET /api/campaigns/{id}/report (structured) and .../report/markdown
(reusing the existing Markdown export path). Adds "可用性/Availability" to the
domain glossary.
Add a thin async loop (run_campaign_loop) that ticks on real wall-clock time,
maps elapsed×time_scale to a window offset via the pure decide_schedule, spawns
due child Runs, and marks the campaign COMPLETED at window end. All authority
lives in the DB (started_at, spawned_indices, status), so the app lifespan can
resume every RUNNING campaign on startup without double-spawning and stop all
loops gracefully on shutdown. A failing plan entry is skipped and recorded
rather than wedging the campaign.
Creating a campaign now starts its loop; POST /api/campaigns/{id}/cancel stops
further spawning (completed child Runs are kept); GET /api/campaigns/{id}
reports live progress (window offset, spawned/completed Run counts).
Add the pure scheduling seam (campaign_scheduler.decide_schedule) that, given a
static plan and window-clock offset, decides which plan entries are due and
whether the window ended — mirroring judgement.combine_case_outcome, with
time_scale confined to the clock mapping so it never touches judgement/report.
The campaign_runner shell maps injected elapsed time to a window offset, spawns
due child Runs through the existing EvalEngine.run(existing_run=...) path with
campaign_id + RunTrigger.CAMPAIGN, and persists spawned-entry indices per entry
for idempotent, restart-recoverable progress. No auto loop yet (ticket 03).
