Skip to main content

​
Funding Partners campaign automation (worker handoff)

The Partners page can now tell an operator whether their Apply for Funding funnel serves from a domain that referral links can actually use, and — when it does and no Funding Partners campaign exists — request the campaign for them.

​
Why there is a worker at all

The CRM’s public Affiliate Manager API is read-only for campaigns. The only verified create path is the ghl_create_affiliate_campaign tool on the local stdio MCP server, which neither Vercel nor Convex can reach. So the operator button does not create anything. It queues one location-scoped intent, and an off-platform worker drives the tool and reports a verified result back.
peek exists because claim has no read-only mode: {"operation":"claim", "limit":0} still leases a request and burns an attempt. During FM-2286 verification exactly that blocked the real worker for ~60s. Use peek to inspect a queue; never claim.

​
Invariants worth preserving

  • The domain gate is server-side. The gate that decides whether a campaign may be queued is recomputed live, from the location’s own funnels, inside requestCampaignCreate; the client cannot assert a domain it does not have. Vendor-owned funnel hosts do not count as connected.
  • Rendering never triggers the live check. getCampaignAutomation serves readiness from the recorded state only (see below). A render that cannot find a usable recording says so — it never presents “not checked” as “not ready”.
  • One intent per location. partnerCampaignRequests.intentKey is locationId + campaign name, so repeated clicks cannot fan out into duplicate campaigns — and the API offers no campaign delete to clean them up.
  • The worker cannot declare success. It forwards the arguments the server composed, including the exact gated page id as stepId, and the raw tool result. classifyMcpCampaignResult requires a verified campaign id; unverified, duplicate-name-ambiguous, or missing results are failures.
  • Completion is claim-fenced. A reclaimed lease increments attempts and a monotonic claimGeneration; completion must echo both. The generation does not reset with an operator retry, so a late worker cannot pass an ABA attempt collision and overwrite the newer run.
  • Campaigns are created as drafts (publish: false). Publishing stays a human decision in the CRM.
  • Every creation rebinds the enrollment workflow (updateEnrollmentWorkflow: true, always). No workflow name is passed — the MCP tool owns the established default.
  • Operator failures are non-technical but distinguishable. Each failure branch persists a stable failureCode (see CampaignRequestFailureCode) and branch-specific operator copy, plus a redacted failureDetail when the tool reported one — but tokens, URLs, file paths, vendor product names, and raw tool errors never reach the request row or operator UI.
  • The manual walkthrough is no longer a path. CampaignSetupOnboarding is not rendered anywhere: setup is in “Partner program setup”, and this automation creates the campaign once step 1’s domain is confirmed. The component file is still in the tree, but nothing mounts it.

​
Adopting a campaign the location already has

Creating is not the only path any more. Sub-accounts that predate the Partner Module arrive with Affiliate Manager campaigns already in them, so convex/prospecting/partnerCampaignSelection.ts lets the operator point the module at one instead of creating a second irreversible campaign.
  • Adoption is explicit or it does not happen. adoptExistingCampaign takes the campaign id the operator clicked. There is no default selection, no “the location has only one campaign so use it”, and no code path that writes a selection without an id. Nothing is pre-selected in the UI either.
  • The id is re-validated server-side. recordCampaignAdoption requires the id to name a campaign under the gated locationId, so a campaign belonging to another tenant is refused even when it points at this location’s funnel.
  • The destination is judged against a page the caller never chooses. recordCampaignAdoption re-classifies the campaign against its pageUrl argument, which it trusts: a caller able to set that argument could name the campaign’s own page and have it accepted. What prevents that is the boundary, not a payload check. The mutation is internal — unreachable from any client — and its only caller is adoptExistingCampaign, which supplies a page a LIVE read just confirmed. Keep both properties or the trust is misplaced; do not make it public on the assumption it self-validates.
  • Adoption is not a way around the duplicate refusal. It writes only a Convex-side selection; enqueueCampaignRequest’s hard refusal on an existing normalized Funding Partners name is untouched.
  • Adoption re-reads the funnel live, for the same reason the create path does: adopting the wrong campaign re-scopes which partners the whole module shows. listExistingCampaigns uses the recording instead, because it feeds an ordinary render.

​
Where the choice is offered, and to whom

PartnerCampaignChoiceCard mounts in the blocked flow on the Partners page — beside the create banner, not inside the “Partner program setup” disclosure, which stays about domain readiness only. Adopt and create are the same decision, so they render as one card; whenever it mounts, the banner is passed suppressCreateView and both are driven off the one offerCampaignChoice boolean, because a page offering two Creates is a page that will get the irreversible one clicked by accident. Both listExistingCampaigns and adoptExistingCampaign are in ADMIN_ONLY_OPERATIONS, so the card mounts only for admins and a plain operator sees today’s create banner unchanged. That leaves a real gap: an operator on a legacy sub-account that already has a usable campaign is offered only the irreversible Create, and has to reach staff to adopt. This is deliberate — the gate is authorization and un-gating it is a human decision, not an implementation detail — not an oversight.

​
Campaign scoping by funnel

convex/lib/partnerCampaignFunnelScope.ts decides which campaigns are the module’s: the ones whose referralRealLink normalizes (host + path, placeholder and query stripped) to the location’s Apply for Funding entry page. When several point at that same funnel, their partners are one deduplicated set, keyed on affiliate id. Partners known only from another funnel’s campaign are excluded. The two directions fail closed opposite ways, on purpose:
  • a campaign is admitted only on positive evidence — an unconfirmed funnel or an unreadable destination is unresolved, never eligible;
  • a partner is hidden only on positive evidence — every campaign we know them to be in resolved to another funnel. Unknown membership, an unplaceable campaign id, or no established scope all keep them visible, because hiding from a guess silently drops somebody’s commissions.
Consumers read the scope through affiliateLinking.campaignFunnelScope, which is built on the same campaignRows page referral attribution uses, so the partner list, the readiness panel, and the resolver cannot disagree about which campaigns the location has.

​
Where domain readiness comes from (FM-2216)

Readiness is established off every render path and replayed on render. The recording is two tables, one row per location either way: partnerFunnelReadinessCache holds a confirmed ready state (funnel, page, connected domain), and partnerFunnelReadinessClears holds the latest not ready classification plus the observation window that orders concurrent writers. A recording older than MAX_READINESS_CACHE_AGE_MS (7 days) is not displayed at all — the card falls back to “not confirmed” rather than reassuring an operator from an answer nothing has re-confirmed. Locations join the cron rotation by having a recording at all, which they get from an operator’s first re-check or create attempt. Nothing enumerates installs: there is no registry of which locations use the Partner Module, and sweeping all of them would spend two internal calls apiece on locations that never open the page. Set MFM_READINESS_REFRESH_ENABLED=false on the deployment to stop the sweep without a deploy. Three display states, kept distinct because they need different responses:
  • ready — the funnel serves from a connected domain.
  • not ready with a named cause (domain-missing, domain-vendor, funnel-missing, page-ambiguous, …) — the operator can fix this in their CRM, and the copy says which thing to fix.
  • could not be confirmed — either nothing has checked yet, or the check itself failed (expired agency credential, upstream outage). This is our problem, not theirs: the copy never tells them to go connect a domain, and it never names a credential, token, path, or vendor.

​
Running the worker

On a machine that can start the MCP server:
At startup the worker runs a preflight before claiming anything: every variable in PARTNER_CAMPAIGN_MCP_REQUIRED_ENV must be set and non-blank, the MCP server must start, and it must offer the campaign-create tool. A worker that fails any of these exits non-zero and names the problem (variables by name only — values are never printed), instead of claiming requests and failing each one with a generic message. Then it proves the app will accept its credential, with one peek (FM-2295). Proving the tool hop said nothing about the queue hop: a stale bearer used to start cleanly and then fail claim with 401 on every pass while the process manager reported a healthy worker. The two failure classes are answered differently, because they mean different things: The probe reads; it never claims. The token appears only in the header and is stripped from any diagnostic text before it is printed. Generate the token with openssl rand -hex 32. Anything shorter than 32 characters is treated as unconfigured, and the route then rejects every request — including a correct-looking one. Set the same value on the app deployment (PARTNER_CAMPAIGN_WORKER_TOKEN); the worker-only variables belong on the worker machine, never in the app’s environment. The token is sent only as an Authorization header and is never logged. The MCP child inherits only the SDK’s safe process variables plus names listed in PARTNER_CAMPAIGN_MCP_ENV_ALLOWLIST. Add only variables the MCP server actually needs. Worker-control variables, including the queue bearer, are rejected and never reach the child process.

​
Repairing enrollment workflows for existing campaigns

Campaigns created before the tool’s cross-location workflow sync existed never had their enrollment workflow rebound. Every claim now carries updateEnrollmentWorkflow: true, so the repair is: give every location that already HAS a Funding Partners campaign a queued request, and let the worker re-run the tool, which finds the existing campaign on its duplicate path and rebinds the workflow (or, if the campaign was renamed away from Funding Partners in the CRM since the last sync, creates one — the same live create-vs-duplicate decision the operator create pipeline makes; see the caveat at the end of this section). Coverage includes locations with no request row at all. The scan is driven from the cached campaigns (ghlAffiliateCampaigns), not the request queue, precisely because the population that matters most — old locations whose campaign predates the request queue entirely — has nothing in partnerCampaignRequests to requeue. For those, the repair INSERTS the request; for locations with an old terminal request it resets it. repairCampaignEnrollmentWorkflows (internal mutation in convex/prospecting/partnerCampaignAutomation.ts) does both. It is never scheduled — no cron, nothing runs on merge — and is invoked explicitly:
Prerequisite: deploy the MCP cross-location workflow-sync fix to the worker machine first. The worker drives the machine-local MCP server, and until that server can bind the enrollment workflow for a location other than its own, a requeued request re-succeeds without repairing anything (or fails and burns attempts). Nothing in this app can verify the worker’s MCP build — confirm it by hand before the first batch. Invocation performs live CRM workflow writes. The mutation itself only writes queue rows, but the worker will then claim each one and drive the real campaign tool against the location’s CRM, rebinding its enrollment workflow. Do not run it against production casually, and use dryRun first. How the arguments behave:
  • notAfter — a fixed epoch-ms cutoff, chosen once for the whole repair (e.g. “now” when you start). It must be strictly before the moment each page runs; the mutation rejects a cutoff at or after now because the termination guarantee below depends on inserted/reset rows landing past it. It separates OLD requests from this repair’s own output: inserted and reset rows get requestedAt: now, past the cutoff, so reruns against the same cutoff skip them as already handled — even after they re-complete — and terminate. A legacy campaign with no request row stays eligible until the real (non-dry) run inserts its request. Do not move the cutoff forward between pages of the same repair.
  • paginationOpts — Convex-native pagination over the campaign cache ({"numItems": N, "cursor": null} to start, then pass continueCursor back). Keep numItems small (~25 campaign rows): the worker drains one claim per pass anyway, and a small page keeps the queued backlog — and the blast radius of a bad worker build — small. Let the queue drain (watch peek depth or the request rows) before the next page. isDone: true means the table is covered.
Only campaigns named exactly Funding Partners — case- and whitespace-sensitive, stricter than the app’s own normalized matching — are processed; anything else in the page is scanned past unwritten. The MCP tool’s sync-enabled duplicate lookup matches that exact string, so a cached row like funding PARTNERS that only the normalized matcher would accept could sail past the tool’s duplicate check and create a second campaign; strict matching keeps a normalized-only row from being mistaken for that evidence. Such normalized-only rows are left for manual handling (rename or adopt in the CRM/app first). Per matched row, the location’s intent request decides what happens:
  • no request row → a queued request is inserted with the canonical intent key, the scanned campaign’s id, and the page/funnel/domain snapshot from the location’s recorded readiness (partnerFunnelReadinessCache). No readiness recording, no insert (skipped.missingPageSnapshot) — the repair never invents the page the tool will bind to; run the location’s re-check first. A recording older than MAX_READINESS_CACHE_AGE_MS (7 days) is also skipped (skipped.staleReadiness): that page is forwarded verbatim as the tool’s stepId, and the rest of the system already stops trusting the cache past that age, so a stale one could rebind the workflow to a superseded page — re-check the location first.
  • terminal (succeeded or failed) at/before the cutoff → reset to queued, the same reset a deliberate operator retry performs — attempts 0, lease/outcome/error/failure fields cleared — preserving the request’s own gated snapshot and recorded ids. A request whose snapshot lost its page is re-snapshotted wholesale from the readiness recording (subject to the same 7-day staleness ceiling → skipped.staleReadiness), or skipped when there is none. A reset that keeps its OWN page never consults the readiness cache. claimGeneration is NOT reset, so a late reply from the original run cannot fence into the repaired one.
  • queued or running at/before the cutoff → skipped (skipped.activeRequest); a live request and its lease are never disturbed.
  • requested after the cutoff → skipped (skipped.alreadyHandled).
Every inserted/reset row carries requestedBy: "system:workflow-repair" as the audit marker. Duplicate Funding Partners rows for one location collapse onto the one intent so one location is never enqueued twice: within a page via an in-memory set, and across pages of a REAL run because page one’s write makes page two’s intent lookup return a requestedAt > notAfter row (skip-handled). A dry run writes nothing, so it only dedupes within a page — cross-page duplicate rows for one location each count again, making preview totals an upper bound. For an exact preview, run the dry run as a single full-table page (numItems ≥ the campaign count). Create-vs-duplicate is the tool’s live decision, not this cache’s. The scanned campaign row is strong EVIDENCE an exactly-named campaign exists — at the last sync it did — so the tool’s live, exact-name duplicate lookup will normally rebind the existing campaign. But upsertSyncedCampaigns never prunes, so the cache can outlive the campaign:
  • Renamed in the CRM — the row keeps its campaign id, so the next sync patches name to the new value and the scan stops matching it. Self-healing.
  • Deleted in the CRM — the campaign stops appearing in any affiliate/commission/payout ref, so nothing ever refreshes the row and it keeps the name Funding Partners indefinitely. This is the durable stale case: the tool finds nothing live and creates a campaign, effectively resurrecting a deliberately deleted one. Before a batch, spot-check that the locations in requeuedLocationIds still have the campaign in GHL (a dryRun gives you that list without writing).
That create is the same one the operator pipeline would make for the location, decided live against current CRM state and never from this cache — the cache only gates WHICH locations get a sync enqueued. The repair deliberately does not gate on campaign syncedAt freshness: campaign rows are derived only from refs embedded in fetched affiliates/commissions/payouts (there is no campaign-list fetch), so a real campaign with no affiliates yet — the primary repair target — never has its syncedAt refreshed, and a freshness gate would skip exactly the locations this backfill exists for. Closing the deleted-campaign case properly requires a repair-only MCP contract that fails instead of creating when an expected campaign id is absent; that is a cross-repo tool-contract change, out of scope here. While the repair request is queued the Partners card changes nothing visible: the campaign exists, so the decision stays none/disabled — same as before the repair.

​
Operating notes

  • A claimed request holds a lease. If the worker dies mid-run, the lease lapses and the request is reclaimed on a later pass, bounded by an attempt ceiling; a max-attempt request becomes failed/retryable, and a deliberate operator retry resets that count.
  • The worker claims exactly one request per pass. MCP initialization, stdio framing, request timeouts, and shutdown are handled by @modelcontextprotocol/sdk.
  • Every pass heartbeats, including an idle one (FM-2295). The singleton partnerCampaignWorkerHeartbeat row records the last successful claim, the last completed pass, the consecutive-failure count, and the last redacted failure reason. An idle pass advances liveness without pretending it claimed work; a refused claim advances the pass stamp but not the claim stamp, which is the stamp staleness is measured from. Without this an idle queue and a dead worker looked identical from the app.
  • A 15-minute cron (alertOnStaleWorkerHeartbeat) posts to Slack when the last successful claim is older than 30 minutes or the failure streak reaches 3. It fires once per outage delivered, not once per check: the outage identity is claimed on the row in the same transaction that decides to alert — which is what stops two overlapping sweeps from both sending — and a clean pass re-arms it. The claim is a lease, not a receipt: only a 2xx from Slack writes alertDeliveredAt, so a failed send is retried on a later sweep instead of burying the outage, and alertAttempts bounds that at 3 so a permanently broken webhook cannot storm. Reading the row during an incident, alertDeliveredAt is the only field that means somebody was actually told. Point it at a channel with PARTNER_CAMPAIGN_WORKER_SLACK_WEBHOOK_URL; unset, the alert no-ops rather than throwing (and records no delivery), and the heartbeat row is still the record of what happened.
  • Failure reasons are logged on the worker machine too, redacted but not reduced to a count — a count told an operator something failed without telling them what, which meant reproducing it by hand to find out.
  • The operator is the fallback alarm. The Slack alerting above is best-effort: a deployment with no webhook configured sends nothing and records nothing as delivered, so nobody is told. The person watching the failed create is the escalation path that always exists, and the Partners banner gives them one. On a failed create — and on one that outruns its budget — the banner keeps the recorded error and the retry, and adds Message us in Slack.
    • The destination is resolved server-side, in getCampaignAutomationState, so the client never decides where an operator is sent. Precedence: the location’s own mapped channel from locationSlackChannels (linked in Admin → Locations), then PARTNER_SUPPORT_SLACK_URL, then https://fundingmachine.slack.com. It always resolves to something.
    • That env var is operator-facing and clickable — a workspace or channel URL, never an incoming-webhook URL. It is unrelated to PARTNER_CAMPAIGN_WORKER_SLACK_WEBHOOK_URL, which is a write-only endpoint for the staleness alert.
    • Most locations resolve to the workspace root today rather than their own channel: at the time of writing only about half the locations with a failed create had a mapped channel, and exactly one mapped row carried a teamId (without one the link uses Slack’s app_redirect form, which resolves the workspace from the operator’s own session). FM-2308 tracks backfilling both.
    • There is deliberately no email route. Support runs in Slack; a mailto would send an operator somewhere nobody is watching for this failure.
  • After a successful create, the Partners page still reads cached campaign rows, so the operator uses the existing Refresh from CRM sync (under their own session) to pull the new campaign in. The worker never syncs on their behalf.
  • If the funnel has no usable domain, nothing is queued: the card explains that a domain must be connected in the CRM first, and offers a re-check.
  • If a card reads “we couldn’t confirm the domain check”, check the agency credential and the sweep’s logs before touching the location — the operator has nothing to fix. An operator re-check is the fastest way to reconfirm once the underlying problem is resolved.