Skip to main content

Underwriting Accuracy — Design

2026-09-02 · Repo: ghl-leadbuilt-marketplace-app-fundingmachine · Surface: /admin/internal/underwriting-review (internal, admin-gated)

Summary

Underwriting Accuracy replaces the verdict-driven review console with an outcome-driven one. Instead of asking a reviewer whether a decision looked right, it asks lenders: for every processing engagement whose Funding plan has settled, what did the engine predict and what actually happened? Those pairs form the Outcome corpus; the console reads the corpus through a three-layer Accuracy scorecard, buckets it by feature (Segment diagnostics), sweeps calibration knobs against it (Knob lab), and calibrates the card catalog against real approvals (Lender calibration). Two ADRs lock the model: Vocabulary is fixed in CONTEXT.md → “Outcome analysis”. This document explains the mechanics behind that vocabulary. It introduces no engine behavior change: RULESET_CODE_VERSION is untouched and the golden snapshots in tests/underwriting.test.ts remain byte-identical.

Non-goals

  • Automatic tuning. The Knob lab ranks candidates and hands a chosen value to the existing calibration-proposal flow (ADR-0001). It never edits constants.
  • Per-lender predictions. The orchestrator makes none, so the lender layer calibrates the catalog rather than scoring a prediction.
  • Revenue Based products (MCA / BLOC / SBA). Excluded until their estimates ship (ADR-0008).
  • Operator-facing docs. This is an internal surface; there are no Mintlify pages.

How the corpus is built

The corpus is materialised server-side into three tables (convex/schemas/analytics.ts):

The build chain

internal.underwritingOutcomes.rebuildCorpus (convex/underwritingOutcomes.ts) is a self-scheduling action. It runs nightly at 03:20 UTC (convex/crons.ts) and on demand from the console’s Rebuild now button (requestCorpusRebuild). Only one build runs at a time: a second request while a live build is running is refused (the cron skips silently, the button reports “already running”). A “running” row older than an hour is treated as a crashed chain and the new build marks it failed/superseded (openBuild). Every build inserts a fresh record set. Nothing is upserted or patched in place, so there is no shared row for two chains to race on: if a superseded chain is somehow still alive it keeps inserting under its own buildId, no reader ever pins a non-complete build, updateBuild ignores its progress patches, and the pruner removes its records.
  1. Pass 1 — completed engagements. Page processingSubmissions in card_processing_complete and then tl_processing_complete, 20 per chunk. For each engagement, load its Funding plan rows, the Contact’s other engagements, the candidate Underwriting Decisions for that Contact in that Location, and the paired credit report, then hand everything to the pure assembler and insertOutcomeRecord the result.
  2. Pass 2 — unsettleable plans. Page fundingPlans rows that have no processing engagement (an Operator who self-processed). They are counted (unsettleablePlans) and never analysed — there is no completion signal to gate on.
  3. Finish. Mark the build complete with its counts, or failed with the error. A failed build leaves the previous corpus untouched. The console bumps a reload token when a build finishes so every loaded tab refetches.
  4. Prune. Completing a build schedules pruneOldBuildRecords, which deletes the record sets of builds that are not the latest complete build, are not running, and finished more than 24 hours ago, 200 records per mutation via by_buildId, rescheduling itself until a chunk comes back short. Storage therefore holds about two record sets at once, and a sweep never outlives the set it pinned.
Every read pins one build. latestCompleteBuildId resolves the newest complete build and every corpus read (listRecords, listStalePending, getRecordDetail, the scorecard / segment / lender aggregations) queries only that build’s records. The read indexes on underwritingOutcomeRecords are all prefixed with buildId (by_build_settled_completedAt, by_build_settled_product_completedAt, by_build_stalePending_completedAt, by_build_processingSubmission) so a query can never see two builds at once, and a build in flight is invisible until it completes. Knob sweeps record the buildId on the sweep row when requested and page that build, so a nightly rebuild mid-sweep cannot move the cursor; until the first build completes, requestKnobSweep refuses with “No completed corpus build to sweep yet” and the other reads return empty. Every step is per-Location: rows are looked up by locationId and a record never joins across tenants.

Assembling one record

convex/lib/outcome/buildOutcomeRecord.ts is pure and unit-tested with fixtures. For one engagement it:
  1. Picks the product from processingType (falling back to the terminal status for legacy rows) — card_stacking or term_loan.
  2. Derives the Outcome from the Funding plan rows that belong to the engagement (rowsForEngagement): the rows stamped with its processingSubmissionId plus the contact’s unlinked rows of the same product that this engagement owns. The two sets are one lender history and are always unioned, because fundingPlans.processingSubmissionId is the billing fee-entitlement link: only rows the processing team enters get it, and only while the engagement is active. Rows an Operator enters never carry it, even on an engagement that also has linked rows, so the unlinked attribution is permanent, not a legacy shim, and there is nothing to backfill. When a contact has several same-product engagements, an unlinked row goes to the latest one submitted before the row was created (else the earliest), so one lender decision is never counted on two records. Row statuses (outcomeDerivation.ts) collapse to approved / declined / pending / not-applied by word match so the card picker, the term-loan picker, and the legacy default picker classify on one rule. A row that references a catalog card inherits the card’s Personal / Business type when it carries none of its own (withCatalogCardType, applied in loadEngagementInputs); without that, every catalog-card approval counted as personal and business-only files scored $0 against their business range. Pending rows are unknown: they count toward pendingCount and set isLowerBound, never toward approvals or declines. Not-applied rows are dropped.
  3. Applies the settle gate (below). Every engagement gets a record with a settleReason; only settled records enter the scorecard.
  4. Pairs the historical Prediction (predictionPairing.ts, ADR-0012 §4): the most recent Decision for that Contact created at or before the engagement was submitted, verified against the range and score snapshot the submission recorded. Submissions stamped with underwritingReportId pair directly. A mismatch, a fall-forward to a post-submission run, or several unverifiable pre-submission runs marks the record pairing-uncertain.
  5. Re-runs the engine over the stored report with default thresholds and now pinned to the pull date, producing the current-ruleset Prediction. When the historical and current ruleset versions match, the reconciliation flag records whether the re-run reproduced the historical headline (status + range). A record that does not reconcile stays in the corpus but is excluded from every layer computed from the current Prediction.
  6. Extracts features (featureExtraction.ts, registry in featureRegistry.ts) in four groups: engine-visible (recomputed through the engine’s own pure modules so they match what the Decision saw), engine-blind (facts on the report the engine ignores today — the discovery target), prediction-derived (what the engine said), and context (product, funding category, pull source, override / custom threshold flags).
Because every build re-assembles every engagement from scratch, a change to assembly or feature logic takes effect on the next rebuild with no version constant to bump.

The settle gate (ADR-0013)

convex/lib/outcome/settleGate.ts. The constants are the gate; they are reported on every scorecard, and changing them is a deliberate decision. The gate is evaluated in order and stops at the first failing condition: The completion mark is completedAt, falling back through cardProcessingCompletedAt, statusChangedAt, and submittedAt for legacy rows. The CRM Client-pipeline stage is a confidence signal only, never a requirement.

Reading the scorecard

convex/lib/outcome/scorecard.ts. Every number on the console shows its sample size n; proportions show a 95% Wilson interval and means a 95% t-interval. The scorecard can be computed under either Prediction kind: Historical (what the operator saw) or Current ruleset re-run.

Layer 1 — Decision accuracy

Did the engine’s predicted class match whether any approval happened?
  • Product statuses collapse to a predicted class: qualified and qualified_conditional → approve; declined → decline; manual review, pending seasoning, not evaluated → abstain. Decisions stored before products[] existed fall back to the legacy card / TL status codes.
  • Accuracy = correct ÷ (approve + decline predictions). Abstains are excluded from the denominator and reported as an abstain rate with their Outcomes shown separately — a high abstain rate is its own finding.
  • A prediction is correct when approve met at least one approved row, or decline met none.

Layer 2 — Amount accuracy

Did total approved dollars land inside the predicted range?
  • Scored only for approve predictions on records that are not pairing-uncertain — a wrong Decision’s range is noise, not error.
  • The range is chosen for the funding category actually pursued: personal range for personal-only files, business range for business-only, combined otherwise; term loans use the TL range. Actual dollars follow the same category.
  • Hit rate = files whose actual landed within [low, high], with below / above counts.
  • Signed error = (actual − midpoint) ÷ midpoint. Negative means the engine over-predicted. The mean signed error with its interval is the bias readout; a mean near zero with a wide interval is noise, an interval that excludes zero is bias.
  • Because Pending rows are unknown, a total with Pending rows is a lower bound (isLowerBound). Silver files therefore skew the amount layer conservatively; filter to Gold to remove that skew.

Layer 3 — Lender calibration (cards only)

convex/lib/outcome/lenderCalibration.ts. The orchestrator predicts nothing per lender, so this layer calibrates the card catalog: per catalog card (or free-text lender), the actual approval rate with interval, Pending count, and median approved limit, against the catalog’s stated minimumScoreRequired and averageLimitProvided.
  • Limit ratio = median approved limit ÷ catalog average limit. Far below 1 means the catalog overstates the card.
  • Below min score = applications where the applicant’s score was under the catalog minimum, and how many of those still approved. Many approvals here mean the stated minimum is too strict.
  • Rows expand to a score-band breakdown so a card can be read the way the engine reads applicants.

Breakdowns

The Scorecard tab also shows the layers by settle tier (Gold / Silver) and by product (0% cards / Term loan). The filter bar scopes everything to a Prediction kind, product, tier, Location, ruleset code, or processing completion-date window.

Segment diagnostics

convex/lib/outcome/segmentDiagnostics.ts. Bucket the current filter set by one feature and report every layer per bucket with n and interval. Numeric features bucket on the registry’s cut points; booleans on yes / no; categoricals on their values; a missing value lands in unknown. Bias flags need the interval to exclude the null, so they are conservative: The Flagged segments list (getFlaggedSegments) runs this across every registry feature and lists each flagged bucket, so the first question — where is the engine systematically wrong? — is answered without picking features one at a time. An engine-blind feature that separates good from bad Outcomes is the primary discovery target: it is a candidate for a new rule.

Knob lab

convex/lib/outcome/knobSweep.ts. Pick a knob from lib/underwriting/knobRegistry.ts, choose candidate values (default: three steps each side of the current value, clamped to the knob’s range), and sweep. When the sweep is requested every candidate is run through the persona guardrail (evaluatePersonaGuardrail in lib/underwriting/batchEval.ts) so it already reports which golden personas would change. The guardrail compares each persona’s decision headline before and after the knob change: reportStatus, the combined / personal-only / business-only / term-loan ranges, the card and term-loan status codes, and each products[] entry’s status, code, and estimate. It deliberately does not compare the serialized results object, whose trace.thresholds echoes every resolved threshold and would mark all 12 personas changed for any threshold knob. The personas run with now pinned to PERSONA_FIXTURE_NOW, the date their bureau dates were written for, so the count does not drift with the wall clock. The sweep also pins the latest complete build at request time, then pages that build’s settled records ten at a time, re-runs the engine under each candidate with now pinned to each record’s pull date, and accumulates a scorecard per candidate. Reading the ranked table:
  • The Δ columns are relative to the knob’s current value (the base row): decision accuracy and amount-in-range rate. |err| gain is how much closer to zero the mean signed error moved; candidates rank by these together.
  • Golden personas (“N would change”) is the count of personas whose decision headline moves under the candidate. It is not a veto; it is the list of decisions a proposal would knowingly change, and the reviewer decides whether each one is intended. The golden snapshots in tests/underwriting.test.ts will still churn on audit fields (ruleset hash, threshold trace) after any calibration; that churn is expected and is not what this column counts.
  • Propose stages the value in the session. The Staged changes panel runs the guardrail for the whole staged set, generates the Cursor prompt and calibration-log entry (buildCalibrationOutput), and submits a calibration proposal (submitCalibration). Each staged value keeps its sweep id and corpus filters; submission rejects a value unless that completed sweep scored it against the currently displayed corpus. Nothing in the lab writes to engine code.

What excluded buckets mean

The Population panel reports every engagement the build scanned, not just what the scorecard uses. Reading it tells you how much ground truth you have and why the rest is missing.

Console surface

app/admin/internal/underwriting-review/page.tsx is the shell; each tab is a component under _components/. Every read goes through the BFF (app/api/underwriting-review/route.ts → postReviewOperation) to actor-gated internal* functions in convex/underwritingOutcomes.ts and convex/underwritingReview.ts. Tabs mount on first visit and stay mounted, so switching tabs never refetches; a finished rebuild bumps one reload token that every loaded tab honours. The console runs inside the CRM iframe at roughly 700px of content width, so tables carry six or seven columns and stack secondary facts under the primary one. The page is built to be read top-down by someone who does not know the vocabulary:
  • Every specialised term is a tooltip. _components/glossary.ts holds one plain-English definition per term (settled, Gold, pairing uncertain, bias, engine-blind, golden personas, …); Term in chrome.tsx renders the dotted underline and the hover card. Copy never re-explains a word inline.
  • Bias, not signed error. A mean signed error is always said as “Over-predicts by 31%” / “Under-predicts by 8%” / “On target” (fmtBias in lib/underwriting/findings.ts); the sign never appears as a symbol. Proportions render with a small interval bar (RateCell, BiasCell) and keep the numeric interval in the cell’s title.
  • Population is one sentence. “132 of 471 completed engagements are scoreable · 93 Gold · 39 Silver”, with stale-pending and last-build facts under it; the eight exclusion counts and the gate definition sit behind a Details toggle.
  • The Prediction kind is a segmented control. Historical vs Current ruleset is the one filter that changes what every number means. Product and tier stay in the primary row; Location, ruleset code and the completion window sit behind “More filters” and show as removable chips when set.
The record drawer (getRecordDetail) is addressed by the processing engagement, not the record: ?record=<processingSubmissionId> in the URL and the same id in the BFF call, resolved within the latest complete build. Record _ids change on every rebuild; the engagement id does not, so a shared link keeps working the next morning. It opens with a one-line verdict — what was predicted, what lenders approved, how it scored — then Prediction vs Outcome for both kinds (the predicted class says “Approve (conditional)” when the product status was qualified_conditional, with the report headline under it), the optional annotation (VerdictControl → upsertLabel; per ADR-0012 it is context for a calibration proposal, never a score), the lender rows by catalog card name, the feature grid collapsed behind a count, and a Process Replay re-run from reconstructUnderwritingInput(replay) with now pinned to the pull date so the replay sees the calendar the engine saw.

Agent access

The Cursor agent reads the corpus through the same actor-gated internal* functions the console uses, with bunx convex run. There is no separate export and no extra code: internal.* functions are only reachable with a deploy key, and requireAdminReviewActor accepts a verifiedAgencyAdmin actor. The CLI addresses a function as file:function, so internal.underwritingOutcomes.getScorecard is underwritingOutcomes:getScorecard on the command line. Records carry credit features (scores, utilization, inquiries, lates), so this is read-only access to production data by an internal agent, not a public export. Treat the output like a credit report: do not paste it into a PR, issue, or chat, and summarise rather than quote. The actor must satisfy reviewActorValidator (convex/underwritingReview.ts): userId is required, verifiedAgencyAdmin: true is what passes the gate, and email is optional but is what requestKnobSweep records as requestedBy, so set it. Filters must satisfy scorecardFiltersValidator (convex/lib/outcome/validators.ts); every field is optional, so {} means the whole latest build. Valid fields: tier (gold / silver / all), product (card_stacking / term_loan / all), locationId, rulesetVersion (the code part, e.g. 2026.08.08), completedFrom / completedTo (epoch ms), predictionKind (historical / current).
requestKnobSweep also accepts "candidates":[680,690,700]; the knob’s current value is always added. Knob ids are the id fields in lib/underwriting/knobRegistry.ts. Without a flag these commands hit the dev deployment in .env.local. Reading production means --prod with the prod deploy key in the environment; that is a deliberate read of live applicant data and nothing else. Never run bunx convex deploy from the agent (AGENTS.md), and never run these against a deployment you have not confirmed with bunx convex env list.

Later

Not in this PR; the triggers are written down so the decision is not re-litigated from scratch.
  • Gate constants are a guess. SETTLE_WINDOW_DAYS = 21 was chosen, not measured. After the first prod build, read the distribution of days between each engagement’s completion mark and the last statusChangedAt on its rows, and set the window from that before trusting the number. The other two gate constants get the same treatment.
  • Workflow component (@convex-dev/workflow). Adopt when a third self-scheduling chain appears alongside rebuildCorpus and runKnobSweep, or when a chain needs per-step retries. Replace both chains at once, not one. The repo mounts no components today, so the first one also buys the convex.config.ts mount and components codegen.
  • Action Cache component (@convex-dev/action-cache). Adopt only when getScorecard / getFlaggedSegments / getLenderCalibration are measurably slow on the real corpus. Key on (buildId, filters); results are immutable per build. Before reaching for it, try storing the unfiltered scorecard on the underwritingOutcomeBuilds row at completion, which needs no component and covers the default console view.

Key files