Skip to main content

​
Lender Outcomes, not reviewer verdicts, are the calibration ground truth

Status: accepted Date: 2026-09-01

​
Decision

  1. The Underwriting orchestrator is calibrated against Outcomes — what lenders actually did on Settled files, read from Funding plan rows — not against human Review verdicts.
  2. The unit of measurement is the Outcome record (one per Settled processing engagement) and the headline is the three-layer Accuracy scorecard: Decision accuracy, Amount accuracy, Lender calibration. Every calibration proposal must cite its effect on the scorecard over the Outcome corpus.
  3. The Underwriting Review Console’s verdict-first flow (pick recent pulls → record a verdict → tweak knobs against the verdicts) is retired. The underwritingReviewLabels and underwritingReviewFlags tables are kept and verdicts remain available as optional annotations on an Outcome record.
  4. Two Predictions are scored per record: the historical Prediction the Operator saw (pinned to its Ruleset version) and a current-ruleset re-run of the stored report with default thresholds. The raw engine decision is scored; Operator product overrides and Location custom thresholds are features, never the thing being scored.
  5. Scope is card stacking and term loan. Revenue Based products join when their estimates exist (ADR-0008).

​
Context

The 2026-05 review console assumed reviewers could judge a Decision’s accuracy by inspection. In practice the console went unused: a verdict is one person’s opinion, it does not scale past a handful of files a week, and it cannot tell the team whether the estimated range was right by 2,000or2,000 or 20,000. Meanwhile the ground truth already existed — every processed file has Funding plan rows with lender approvals, declines, and approved amounts — but was only consulted as an ungated per-contact sum (sumFundedPlans). The orchestrator emits no per-lender prediction. The approvalProbability and related fields on old fundingPlans rows are artefacts of a retired AI plan generator (LEGACY_GENERATED_PLAN_FIELDS), so a “lender hit rate” against an engine probability is not measurable. The third scorecard layer therefore calibrates the card catalog’s stated minimumScoreRequired and averageLimitProvided against actual approvals instead. Term-loan Outcomes come from TL Funding plan rows only. Engine by MoneyLion marketplace data (offers, derivedStatus, per-lender events) shows pre-qualification and application events but never an accepted offer, so it is at most a feature.

​
Considered options

  • Keep verdicts as ground truth and add the funded comparison as context — rejected. Keeps the unscalable human step in the loop and leaves the dollar error unmeasured.
  • Score the effective decision (override ?? engine) — rejected. Overrides hide engine error; we want to see where Operators had to correct the engine.
  • Score only the historical Prediction — rejected. Ruleset bumps would make files incomparable; the current-ruleset re-run lets one build answer “would today’s engine have been right on every file we have ever processed”.
  • Outcome as ground truth with verdicts demoted to annotations (accepted).

​
Consequences

  • The review console page is rebuilt around the Outcome corpus (Scorecard → Segment diagnostics → Knob lab, with the process replay kept as the single-record drill-down). The pure libraries beneath it — knob registry, what-if override seam (ADR-0001), persona guardrail, input assembly, calibration proposal → generated prompt — are reused unchanged.
  • The Knob lab never edits engine constants or thresholds. A chosen change still ships as a code edit with a RULESET_CODE_VERSION bump and golden-snapshot review, exactly as ADR-0001 requires.
  • Records whose same-ruleset re-run does not reproduce the historical headline carry a reconciliation flag and are excluded from re-run-based layers. Closing that gap forward requires persisting incomeType, monthlyBankDeposits, and businessDeposits on underwritingResults (additive optional fields).
  • Glossary: CONTEXT.md → “Outcome analysis” section, and the revised “Underwriting review” / “Review verdict” entries. Companion decision on which files count: ADR-0013.