Underwriting Accuracy — Design
2026-09-02 · Repo:ghl-leadbuilt-marketplace-app-fundingmachine · Surface: /admin/internal/underwriting-review (internal, admin-gated)
Summary
Underwriting Accuracy replaces the verdict-driven review console with an outcome-driven one. Instead of asking a reviewer whether a decision looked right, it asks lenders: for every processing engagement whose Funding plan has settled, what did the engine predict and what actually happened? Those pairs form the Outcome corpus; the console reads the corpus through a three-layer Accuracy scorecard, buckets it by feature (Segment diagnostics), sweeps calibration knobs against it (Knob lab), and calibrates the card catalog against real approvals (Lender calibration). Two ADRs lock the model:- ADR-0012 — Outcome as calibration ground truth: lender outcomes are the ground truth; reviewer verdicts are optional annotations, never a score.
- ADR-0013 — Settled-file gate: which engagements are trustworthy enough to score, and when.
CONTEXT.md → “Outcome analysis”. This document
explains the mechanics behind that vocabulary. It introduces no engine
behavior change: RULESET_CODE_VERSION is untouched and the golden
snapshots in tests/underwriting.test.ts remain byte-identical.
Non-goals
- Automatic tuning. The Knob lab ranks candidates and hands a chosen value to the existing calibration-proposal flow (ADR-0001). It never edits constants.
- Per-lender predictions. The orchestrator makes none, so the lender layer calibrates the catalog rather than scoring a prediction.
- Revenue Based products (MCA / BLOC / SBA). Excluded until their estimates ship (ADR-0008).
- Operator-facing docs. This is an internal surface; there are no Mintlify pages.
How the corpus is built
The corpus is materialised server-side into three tables (convex/schemas/analytics.ts):
The build chain
internal.underwritingOutcomes.rebuildCorpus (convex/underwritingOutcomes.ts)
is a self-scheduling action. It runs nightly at 03:20 UTC (convex/crons.ts)
and on demand from the console’s Rebuild now button
(requestCorpusRebuild). Only one build runs at a time: a second request
while a live build is running is refused (the cron skips silently, the button
reports “already running”). A “running” row older than an hour is treated as a
crashed chain and the new build marks it failed/superseded (openBuild).
Every build inserts a fresh record set. Nothing is upserted or patched in
place, so there is no shared row for two chains to race on: if a superseded
chain is somehow still alive it keeps inserting under its own buildId, no
reader ever pins a non-complete build, updateBuild ignores its progress
patches, and the pruner removes its records.
- Pass 1 — completed engagements. Page
processingSubmissionsincard_processing_completeand thentl_processing_complete, 20 per chunk. For each engagement, load its Funding plan rows, the Contact’s other engagements, the candidate Underwriting Decisions for that Contact in that Location, and the paired credit report, then hand everything to the pure assembler andinsertOutcomeRecordthe result. - Pass 2 — unsettleable plans. Page
fundingPlansrows that have no processing engagement (an Operator who self-processed). They are counted (unsettleablePlans) and never analysed — there is no completion signal to gate on. - Finish. Mark the build
completewith its counts, orfailedwith the error. A failed build leaves the previous corpus untouched. The console bumps a reload token when a build finishes so every loaded tab refetches. - Prune. Completing a build schedules
pruneOldBuildRecords, which deletes the record sets of builds that are not the latest complete build, are notrunning, and finished more than 24 hours ago, 200 records per mutation viaby_buildId, rescheduling itself until a chunk comes back short. Storage therefore holds about two record sets at once, and a sweep never outlives the set it pinned.
latestCompleteBuildId resolves the newest
complete build and every corpus read (listRecords, listStalePending,
getRecordDetail, the scorecard / segment / lender aggregations) queries only
that build’s records. The read indexes on underwritingOutcomeRecords are all
prefixed with buildId (by_build_settled_completedAt,
by_build_settled_product_completedAt, by_build_stalePending_completedAt,
by_build_processingSubmission) so a query can never see two builds at once,
and a build in flight is invisible until it completes. Knob sweeps record the
buildId on the sweep row when requested and page that build, so a nightly
rebuild mid-sweep cannot move the cursor; until the first build completes,
requestKnobSweep refuses with “No completed corpus build to sweep yet” and
the other reads return empty.
Every step is per-Location: rows are looked up by locationId and a record
never joins across tenants.
Assembling one record
convex/lib/outcome/buildOutcomeRecord.ts is pure and unit-tested with
fixtures. For one engagement it:
- Picks the product from
processingType(falling back to the terminal status for legacy rows) —card_stackingorterm_loan. - Derives the Outcome from the Funding plan rows that belong to the
engagement (
rowsForEngagement): the rows stamped with itsprocessingSubmissionIdplus the contact’s unlinked rows of the same product that this engagement owns. The two sets are one lender history and are always unioned, becausefundingPlans.processingSubmissionIdis the billing fee-entitlement link: only rows the processing team enters get it, and only while the engagement is active. Rows an Operator enters never carry it, even on an engagement that also has linked rows, so the unlinked attribution is permanent, not a legacy shim, and there is nothing to backfill. When a contact has several same-product engagements, an unlinked row goes to the latest one submitted before the row was created (else the earliest), so one lender decision is never counted on two records. Row statuses (outcomeDerivation.ts) collapse to approved / declined / pending / not-applied by word match so the card picker, the term-loan picker, and the legacy default picker classify on one rule. A row that references a catalog card inherits the card’s Personal / Business type when it carries none of its own (withCatalogCardType, applied inloadEngagementInputs); without that, every catalog-card approval counted as personal and business-only files scored $0 against their business range. Pending rows are unknown: they count towardpendingCountand setisLowerBound, never toward approvals or declines. Not-applied rows are dropped. - Applies the settle gate (below). Every engagement gets a record with a
settleReason; onlysettledrecords enter the scorecard. - Pairs the historical Prediction (
predictionPairing.ts, ADR-0012 §4): the most recent Decision for that Contact created at or before the engagement was submitted, verified against the range and score snapshot the submission recorded. Submissions stamped withunderwritingReportIdpair directly. A mismatch, a fall-forward to a post-submission run, or several unverifiable pre-submission runs marks the record pairing-uncertain. - Re-runs the engine over the stored report with default thresholds and
nowpinned to the pull date, producing the current-ruleset Prediction. When the historical and current ruleset versions match, the reconciliation flag records whether the re-run reproduced the historical headline (status + range). A record that does not reconcile stays in the corpus but is excluded from every layer computed from the current Prediction. - Extracts features (
featureExtraction.ts, registry infeatureRegistry.ts) in four groups: engine-visible (recomputed through the engine’s own pure modules so they match what the Decision saw), engine-blind (facts on the report the engine ignores today — the discovery target), prediction-derived (what the engine said), and context (product, funding category, pull source, override / custom threshold flags).
The settle gate (ADR-0013)
convex/lib/outcome/settleGate.ts. The constants are the gate; they are
reported on every scorecard, and changing them is a deliberate decision.
The gate is evaluated in order and stops at the first failing condition:
The completion mark is
completedAt, falling back through
cardProcessingCompletedAt, statusChangedAt, and submittedAt for legacy
rows. The CRM Client-pipeline stage is a confidence signal only, never a
requirement.
Reading the scorecard
convex/lib/outcome/scorecard.ts. Every number on the console shows its
sample size n; proportions show a 95% Wilson interval and means a 95%
t-interval. The scorecard can be computed under either Prediction kind:
Historical (what the operator saw) or Current ruleset re-run.
Layer 1 — Decision accuracy
Did the engine’s predicted class match whether any approval happened?- Product statuses collapse to a predicted class:
qualifiedandqualified_conditional→ approve;declined→ decline; manual review, pending seasoning, not evaluated → abstain. Decisions stored beforeproducts[]existed fall back to the legacy card / TL status codes. - Accuracy = correct ÷ (approve + decline predictions). Abstains are excluded from the denominator and reported as an abstain rate with their Outcomes shown separately — a high abstain rate is its own finding.
- A prediction is correct when
approvemet at least one approved row, ordeclinemet none.
Layer 2 — Amount accuracy
Did total approved dollars land inside the predicted range?- Scored only for approve predictions on records that are not pairing-uncertain — a wrong Decision’s range is noise, not error.
- The range is chosen for the funding category actually pursued: personal range for personal-only files, business range for business-only, combined otherwise; term loans use the TL range. Actual dollars follow the same category.
- Hit rate = files whose actual landed within
[low, high], with below / above counts. - Signed error = (actual − midpoint) ÷ midpoint. Negative means the engine over-predicted. The mean signed error with its interval is the bias readout; a mean near zero with a wide interval is noise, an interval that excludes zero is bias.
- Because Pending rows are unknown, a total with Pending rows is a lower
bound (
isLowerBound). Silver files therefore skew the amount layer conservatively; filter to Gold to remove that skew.
Layer 3 — Lender calibration (cards only)
convex/lib/outcome/lenderCalibration.ts. The orchestrator predicts nothing
per lender, so this layer calibrates the card catalog: per catalog card
(or free-text lender), the actual approval rate with interval, Pending count,
and median approved limit, against the catalog’s stated minimumScoreRequired
and averageLimitProvided.
- Limit ratio = median approved limit ÷ catalog average limit. Far below 1 means the catalog overstates the card.
- Below min score = applications where the applicant’s score was under the catalog minimum, and how many of those still approved. Many approvals here mean the stated minimum is too strict.
- Rows expand to a score-band breakdown so a card can be read the way the engine reads applicants.
Breakdowns
The Scorecard tab also shows the layers by settle tier (Gold / Silver) and by product (0% cards / Term loan). The filter bar scopes everything to a Prediction kind, product, tier, Location, ruleset code, or processing completion-date window.Segment diagnostics
convex/lib/outcome/segmentDiagnostics.ts. Bucket the current filter set by
one feature and report every layer per bucket with n and interval. Numeric
features bucket on the registry’s cut points; booleans on yes / no;
categoricals on their values; a missing value lands in unknown.
Bias flags need the interval to exclude the null, so they are conservative:
The Flagged segments list (
getFlaggedSegments) runs this across every
registry feature and lists each flagged bucket, so the first question — where
is the engine systematically wrong? — is answered without picking features one
at a time. An engine-blind feature that separates good from bad Outcomes is the
primary discovery target: it is a candidate for a new rule.
Knob lab
convex/lib/outcome/knobSweep.ts. Pick a knob from
lib/underwriting/knobRegistry.ts, choose candidate values (default: three
steps each side of the current value, clamped to the knob’s range), and sweep.
When the sweep is requested every candidate is run through the persona
guardrail (evaluatePersonaGuardrail in lib/underwriting/batchEval.ts) so
it already reports which golden personas would change. The guardrail compares
each persona’s decision headline before and after the knob change:
reportStatus, the combined / personal-only / business-only / term-loan
ranges, the card and term-loan status codes, and each products[] entry’s
status, code, and estimate. It deliberately does not compare the serialized
results object, whose trace.thresholds echoes every resolved threshold and
would mark all 12 personas changed for any threshold knob. The personas run
with now pinned to PERSONA_FIXTURE_NOW, the date their bureau dates were
written for, so the count does not drift with the wall clock. The sweep also pins
the latest complete build at request time, then pages that build’s settled
records ten at a time, re-runs the engine under each candidate with now
pinned to each record’s pull date, and accumulates a scorecard per candidate.
Reading the ranked table:
- The Δ columns are relative to the knob’s current value (the
baserow): decision accuracy and amount-in-range rate. |err| gain is how much closer to zero the mean signed error moved; candidates rank by these together. - Golden personas (“N would change”) is the count of personas whose
decision headline moves under the candidate. It is not a veto; it is the
list of decisions a proposal would knowingly change, and the reviewer decides
whether each one is intended. The golden snapshots in
tests/underwriting.test.tswill still churn on audit fields (ruleset hash, threshold trace) after any calibration; that churn is expected and is not what this column counts. - Propose stages the value in the session. The Staged changes panel
runs the guardrail for the whole staged set, generates the Cursor prompt and
calibration-log entry (
buildCalibrationOutput), and submits a calibration proposal (submitCalibration). Each staged value keeps its sweep id and corpus filters; submission rejects a value unless that completed sweep scored it against the currently displayed corpus. Nothing in the lab writes to engine code.
What excluded buckets mean
The Population panel reports every engagement the build scanned, not just what the scorecard uses. Reading it tells you how much ground truth you have and why the rest is missing.Console surface
app/admin/internal/underwriting-review/page.tsx is the shell; each tab is a
component under _components/. Every read goes through the BFF
(app/api/underwriting-review/route.ts → postReviewOperation) to
actor-gated internal* functions in convex/underwritingOutcomes.ts and
convex/underwritingReview.ts. Tabs mount on first visit and stay mounted, so
switching tabs never refetches; a finished rebuild bumps one reload token that
every loaded tab honours. The console runs inside the CRM iframe at roughly
700px of content width, so tables carry six or seven columns and stack
secondary facts under the primary one.
The page is built to be read top-down by someone who does not know the
vocabulary:
- Every specialised term is a tooltip.
_components/glossary.tsholds one plain-English definition per term (settled, Gold, pairing uncertain, bias, engine-blind, golden personas, …);Terminchrome.tsxrenders the dotted underline and the hover card. Copy never re-explains a word inline. - Bias, not signed error. A mean signed error is always said as
“Over-predicts by 31%” / “Under-predicts by 8%” / “On target” (
fmtBiasinlib/underwriting/findings.ts); the sign never appears as a symbol. Proportions render with a small interval bar (RateCell,BiasCell) and keep the numeric interval in the cell’s title. - Population is one sentence. “132 of 471 completed engagements are scoreable · 93 Gold · 39 Silver”, with stale-pending and last-build facts under it; the eight exclusion counts and the gate definition sit behind a Details toggle.
- The Prediction kind is a segmented control. Historical vs Current ruleset is the one filter that changes what every number means. Product and tier stay in the primary row; Location, ruleset code and the completion window sit behind “More filters” and show as removable chips when set.
The record drawer (
getRecordDetail) is addressed by the processing
engagement, not the record: ?record=<processingSubmissionId> in the URL and
the same id in the BFF call, resolved within the latest complete build. Record
_ids change on every rebuild; the engagement id does not, so a shared link
keeps working the next morning. It opens with a one-line verdict — what was
predicted, what lenders approved, how it scored — then Prediction vs Outcome
for both kinds (the predicted class says “Approve (conditional)” when the
product status was qualified_conditional, with the report headline under
it), the optional annotation (VerdictControl → upsertLabel; per ADR-0012 it
is context for a calibration proposal, never a score), the lender rows by
catalog card name, the feature grid collapsed behind a count, and a Process
Replay re-run from reconstructUnderwritingInput(replay) with now pinned
to the pull date so the replay sees the calendar the engine saw.
Agent access
The Cursor agent reads the corpus through the same actor-gatedinternal*
functions the console uses, with bunx convex run. There is no separate export
and no extra code: internal.* functions are only reachable with a deploy key,
and requireAdminReviewActor accepts a verifiedAgencyAdmin actor. The CLI
addresses a function as file:function, so internal.underwritingOutcomes.getScorecard
is underwritingOutcomes:getScorecard on the command line.
Records carry credit features (scores, utilization, inquiries, lates), so this
is read-only access to production data by an internal agent, not a public
export. Treat the output like a credit report: do not paste it into a PR,
issue, or chat, and summarise rather than quote.
The actor must satisfy reviewActorValidator (convex/underwritingReview.ts):
userId is required, verifiedAgencyAdmin: true is what passes the gate, and
email is optional but is what requestKnobSweep records as requestedBy, so
set it. Filters must satisfy scorecardFiltersValidator
(convex/lib/outcome/validators.ts); every field is optional, so {} means the
whole latest build. Valid fields: tier (gold / silver / all), product
(card_stacking / term_loan / all), locationId, rulesetVersion
(the code part, e.g. 2026.08.08), completedFrom / completedTo
(epoch ms), predictionKind (historical / current).
requestKnobSweep also accepts "candidates":[680,690,700]; the knob’s
current value is always added. Knob ids are the id fields in
lib/underwriting/knobRegistry.ts.
Without a flag these commands hit the dev deployment in .env.local. Reading
production means --prod with the prod deploy key in the environment; that is
a deliberate read of live applicant data and nothing else. Never run
bunx convex deploy from the agent (AGENTS.md), and never run these against
a deployment you have not confirmed with bunx convex env list.
Later
Not in this PR; the triggers are written down so the decision is not re-litigated from scratch.- Gate constants are a guess.
SETTLE_WINDOW_DAYS = 21was chosen, not measured. After the first prod build, read the distribution of days between each engagement’s completion mark and the laststatusChangedAton its rows, and set the window from that before trusting the number. The other two gate constants get the same treatment. - Workflow component (
@convex-dev/workflow). Adopt when a third self-scheduling chain appears alongsiderebuildCorpusandrunKnobSweep, or when a chain needs per-step retries. Replace both chains at once, not one. The repo mounts no components today, so the first one also buys theconvex.config.tsmount andcomponentscodegen. - Action Cache component (
@convex-dev/action-cache). Adopt only whengetScorecard/getFlaggedSegments/getLenderCalibrationare measurably slow on the real corpus. Key on(buildId, filters); results are immutable per build. Before reaching for it, try storing the unfiltered scorecard on theunderwritingOutcomeBuildsrow at completion, which needs no component and covers the default console view.

