Wrap the judge prompt and two docstrings past the 120-col convention;
record three implementation rulings in the v0.9 spec (exploration read
outlets, round-based sampling, findings carrying all ratings).
The drawer hand-rolled chat bubbles duplicating ChatBubble's shape.
Add a mirrored layout prop (virtual user left, target right) and reuse
the shared component.
report, markdown and analysis each repeated the
summarize_exploration(repo.list_by_campaign(...)) shape; collapse it
into summarize_campaign_exploration so the aggregation has one home.
Ticket 05 asks the judge review conclusions to flow into the report
verbatim; the aggregation silently dropped good/acceptable dimensions.
Collect every finding sorted poor-first and color drawer tags by rating.
E2E on t480 showed assistant bubbles rendering {'content': '...'} because
tutu replies carry msgBody as a parsed object and the router stored
str(reply.content). Coerce to the inner text before persisting.
Exploration sessions aggregate into a single exploration summary
(session counts, goal-achievement rate, issue lists from experience
records, judge conclusions when reviewed) that feeds three exits:
the campaign report gains an exploration dimension, the v0.7 analysis
stage-two input gains the summary (stats only, never full dialogues),
and the Markdown export appends a findings appendix after analysis and
comparison. With no exploration data every output stays unchanged.
Adds unit tests for the campaign report beyond value correctness:
cancellation exclusion from denominators (ADR-0004), window clamping,
naive/aware timestamp equivalence, plus schema validation of the report
dict (key sets, types, rate bounds, contiguous buckets, capability sort,
ISO-UTC timestamps) and consistency with the single aggregate_runs seam.
After an exploration session closes, the platform samples up to 3
conversation rounds and runs an independent judge-role review through
the v0.7 ChatClient seam, persisting quality-dimension conclusions
(attitude, professionalism, hallucination) into the session's
judge_review. The review runs as a background task: failures are
recorded without touching session state or the first-hand experience
record, and a missing model config skips silently.
Resident agents call GET /api/exploration/patrol once per cycle to see
every running production-line campaign that opted into exploration
(seed set present), the new results since the last watermark (reusing
campaign report aggregation), and the remaining exploration budget.
The watermark advances after each call so subsequent calls only report
increments; accelerated and terminal campaigns are excluded.
v0.9 ticket 02. Campaigns now carry an exploration seed set (seed
personas × seed goals — the comparability unit for exploratory
evaluation) and an optional budget override, stored as JSON columns
isomorphic to plan. Empty seeds normalize to null, marking the campaign
as opted out of exploration. resolve_budget merges per-field overrides
into platform defaults; enforcement stays server-side. The create form
gains seed lists and budget inputs (minutes → seconds), submitting null
when left empty.
v0.9 ticket 01. Independent exploration_sessions/exploration_messages
entities (never merged into EvalRun, keeping ADR-0001/0002 semantics
intact): create/message/close APIs forward virtual-user messages through
the target's real channel, persist both parties' rows with latency, and
close with a whitelist-normalized experience record. Budget enforcement
is a platform ledger — sessions per window, turns per session, and
session interval overruns return 409 with readable reasons; accelerated
lines accept manual sessions only. Messages delivered but unanswered
still consume a turn so timeouts cannot bypass the budget.
Record the Campaign milestone (v0.6: durable scheduler, dual-axis
report, campaigns page) and the campaign-intelligence milestone (v0.7:
two-phase analysis agent, analysis model default/override, drawer
section, auto-trigger and markdown export). Bump the single-source
version to 0.7.0.
The scheduler loop enqueues the analysis task when a realtime campaign
completes; accelerated or cancelled campaigns and a missing analysis
model skip silently. The campaign markdown export appends the analysis
appendix (overall, problems, narratives, suggestions) when a completed
analysis exists.
Render the campaign analysis in the report drawer: status row for
generating (5s polling), failed (error + retry) and empty states, then
the structured result — overall callout, problem cards with severity
tags and evidence chips linking to run reports, per-scenario narratives
and priority-sorted suggestions. Generate buttons are terminal-only with
guidance when no analysis model is configured.
Add the analysis role's execution path: a two-phase orchestration
(per-scenario diagnosis gathered in parallel, then a synthesis pass)
that reads the existing campaign report aggregation plus capped failure
samples, validates the LLM's JSON against the report schema, and strips
fabricated run/scenario references before persisting. Results upsert one
row per campaign (generating/completed/failed) with the model config
snapshot; GET/POST /api/campaigns/{id}/analysis expose the state machine,
guarding non-terminal campaigns and missing analysis models with 400s.
Campaigns can pin an analysis model config instead of following the
global analysis default. Creation validates the referenced config
exists (400 otherwise); the create form offers enabled chat configs
with the global default as the fallback option.
Introduce ModelPurpose.ANALYSIS and a globally-unique is_analysis_default
marker on chat model configs so campaign analysis can resolve its model.
Service rejects disabled or non-chat configs; repo clears the previous
holder on set. Documented the analysis role in CONTEXT.md.
Report drawer widens to 1040px with a four-card metric row (rate-graded
colours), the embedded scenario-lane process timeline with now-line,
a dual-series pass-rate/availability trend next to a horizontal
capability ranking bar chart, and a filterable/sortable sub-run table
with a latency column. Chart configs move to the charts v2 scale/axis
API — the old yAxis key was dead v1 config.
Plan preview markers are plain colour blocks (names live in the
tooltip and legend); blocks that would overlap spread onto staggered
rows. Plan start time is limited to the window length via InputNumber
max plus a validator, re-checked immediately when the window shrinks.
Create surface moves from a 640px modal to a 920px two-column drawer:
basic info + time/speed on the left, plan preview and a grid-aligned
plan editor (headers, searchable scenario selects, scrollable entries)
on the right, so many-entry plans stay editable. The expanded-row
process timeline switches from a single crowded axis to per-scenario
lanes with a now-line for active campaigns.
Each campaign row expands to a per-Run process timeline: on expand it fetches
GET /campaigns/{id}/timeline and places each child Run on the shared
WindowTimeline by accelerated window offset, coloured by run status, with
scenario/pass-rate/latency tooltips and click-through to the run report.
Running campaigns refresh on the existing 5s poll; terminal ones fetch once.
WindowTimeline gains a colorMap prop for semantic status colours. (v0.6 ticket 10)
Above the plan editor, adapt each plan entry (offset, scenario, count) into
WindowTimeline markers coloured per scenario with a count badge, re-rendering
live as the form changes via Form.useWatch. Empty plan shows a placeholder.
Purely form-local, no backend change. (v0.6 ticket 09)
A domain-agnostic horizontal timeline: an axis over the service-cycle window
with offset-positioned markers, stable per-colorKey colours, optional badges,
hover tooltips, click callbacks, and an optional legend. Shared base for the
plan preview (09) and process timeline (10). (v0.6 ticket 08)
Users now pick a window length + "how long it should actually take" and the
form derives time_scale (window ÷ target), showing 倍速 ×N and 加速后耗时
read-only; a 实时 switch pins real wall-clock (×1). Validation blocks a target
longer than the window. List/detail display accelerated duration instead of raw
×N. POST /campaigns contract unchanged. (v0.6 ticket 07)
build_campaign_timeline flattens a campaign's child Runs into offset-sorted
per-Run entries (distinct from the report's 12-bucket aggregation), reusing a
shared _run_window_offset口径 so both views place a run identically. Exposes
GET /campaigns/{id}/timeline and the api.ts type/call. (v0.6 ticket 06)
Campaign.summary was a bare Optional[dict] while RunSummary is a typed VO —
scheduler state (spawned_indices/errors) flowed untyped through
set_/get_summary. Introduce CampaignSummary + SchedulerState (extra=allow,
validate_assignment), mirroring RunSummary; campaign_runner reads/writes the
VO. Also converge the ~10 repeated JSON column get/set pairs onto
_json_dumps/_json_loads helpers, unifying ensure_ascii=False and fixing the
set_modalities ensure_ascii=True trap.
Three near-identical Turn(...)+save_turn blocks (send-fail / poll-except /
happy path) collapse into one _persist_turn helper differing only by the
optional fields set. The期望→隐式规则 translation becomes a pure,
unit-testable derive_implicit_rules seam (CONTEXT: 期望与规则叠加生效),
and the rule_type→ModelPurpose map is hoisted to a module constant.
Single-run summary口径 (pass_rate / judged_pass_rate / avg_latency /
connectivity split) was inlined in run(), reachable only by driving a
whole async run, and report.py recomputed judged_pass_rate independently.
Extract build_run_summary — a pure function parallel to aggregate_runs
(cross-run) and combine_case_outcome (case-level). run() now collects
material and delegates; judged_pass_rate is stored in RunSummary so the
report reads it instead of recomputing.
Read paths recomputed per-case pass/connectivity independently — report
generation, the logs endpoint, and the frontend each derived it, and the
frontend's every(passed) recompute ignored the engine's authoritative
verdict. Extract resolve_case_verdicts: a single pure seam that prefers
stored case_outcomes verbatim and approximates only for legacy runs. The
logs endpoint now surfaces case_verdicts so the frontend reads instead of
recomputing.
Seven pages repeated the same load-on-mount + loading + try/finally +
reload-button skeleton, each re-implementing tab-active refresh, silent
polling, and (in two places) a hand-rolled requestId race guard. Extract two
composable hooks: useResource(fetcher, {tabPath, deps}) owning data/loading/
reload with a built-in race guard and auto tab-active refresh, and
usePolling(fn, ms, enabled) replacing the hand-written setInterval effects.
Migrate all seven pages onto them; Targets/Scenarios/ModelConfigs also gain a
uniform tab-active refresh they previously lacked. Verified via tsc --noEmit
and npm run build (no frontend test runner exists).
report.py mixed DB-reading generation with string formatting: the four
render_*_report(run_id, session) functions each re-fetched via
generate_report, so the HTML/Markdown/JSON formatting was welded to storage
and could not be unit-tested from a plain dict. Extract the formatting into a
new pure report_render module whose renderers take the already-built report
dict (no session, no storage import). Migrate every caller to generate-then-
render, delete the old coupled renderers with no back-compat shim, and drop
the _aggregate_runs middle-man alias in favour of metrics.aggregate_runs.
