September 2026
A utility often has several programs that a customer could join. A propensity model can score each eligible program and recommend the one with the largest predicted enrollment probability. This is a sensible baseline, but it can miss local patterns: the model may repeatedly overpredict or underpredict a program for customers with similar behavior.
We study a simple response. First, compress each customer’s behavioral history into an embedding. Second, estimate a propensity for every program. Third, find historical customers who look like the target customer and use their out-of-fold prediction errors to make a local correction. The central difficulty is that a more detailed match leaves fewer peers. A useful recommender must balance similarity against support.
The evidence gives a clear division of labor. Behavioral embeddings provide the main predictive gain. Neighborhood correction provides a small improvement in propensity calibration but no confirmed improvement in the winning recommendation. The theory remains useful because it states when local evidence is precise enough to trust and when the system should keep the base result or abstain.
Figure 1. The embedding supplies the predictive representation. The neighborhood supplies local evidence and a measure of its reliability.
Let \(x\) denote a customer and let \(\cA=\{a_1,\ldots,a_K\}\) be a catalog of \(K\) programs. Eligibility leaves the feasible set \(\cA(x)\subseteq\cA\). For program \(a\), define the true enrollment propensity
The vector \(p(x)=(p_{a_1}(x),\ldots,p_{a_K}(x))\) is the customer’s true propensity row. For customers \(x_1,\ldots,x_N\), stacking these rows gives the true propensity matrix \(P\in[0,1]^{N\times K}\). A fitted model produces rows \(\widetilde p(x_i)\) and the fitted matrix \(\widetilde P\). The direct recommender chooses
Behavior provides useful customer coordinates. Let \(H\in\mathbb R^{N\times J}\) record the pre-decision activity of \(N\) customers over \(J\) items or events. A rank-\(D\) factorization gives
Here \((\cdot)_{i\cdot}\) selects matrix row \(i\), \({}^\top\) denotes transpose, and \(z(x_i)\) is customer \(i\)’s behavioral embedding. If \(s(x)\) is a vector of ordinary customer attributes, a program model has the form
This transparent factorization follows the collaborative-filtering tradition [1]; richer sequential representations could be used instead [2, 3]. What matters is whether nearby coordinates identify customers with similar future propensities or similar remaining model errors.
Let \(x_1,\ldots,x_n\) be historical reference customers. Divide the available customer representation into \(M\) ordered blocks. After the first \(m\) blocks, let \(d_m(x,x_i)\) be the cumulative distance from target \(x\) to reference customer \(x_i\). We require
For a fixed radius \(r\ge0\), define the level-\(m\) neighborhood and its size by
Only nonempty neighborhoods are used.
Proposition 1 (More detail leaves fewer peers). For \(m=1,\ldots,M-1\),
Proof. If \(d_{m+1}(x,x_i)\le r\), then \(d_m(x,x_i)\le d_{m+1}(x,x_i)\le r\). Every later-level peer was therefore already an earlier-level peer.□
To measure the quality of the remaining match, let \(g(x)=(g_a(x):a\in\cA)\) be any program-indexed target. Define
For direct peer averaging, take \(g=p\). For residual correction, take
Thus \(\rho(x)\) is the error left by the base propensity model.
Proposition 2 (Refinement improves worst-case local similarity). If \(\cN_{m+1}(x;r)\) is nonempty, then
Proof. The later neighborhood is a subset of the earlier one by Proposition 1. The maximum of the same nonnegative quantity cannot increase when points are removed.□
These propositions describe the basic tension. Adding detail cannot worsen the worst match among the peers who remain, but it cannot add support. Propensity rows themselves provide a convenient geometry, for example
A leave-one-program-out distance can omit coordinate \(a\) when calibrating program \(a\), although the experiments find that the omitted coordinate often contains useful retrieval information.
For peer \(i\) and eligible program \(a\), let \(Y_{ia}\in[0,B]\) be a bounded calibration outcome with \(\E[Y_{ia}\mid x_i]=p_a(x_i)\). Binary enrollment has \(B=1\). Reference predictions must be out of fold or produced by a separately fitted model. At level \(m\), correct the target score with the peers’ mean residual:
Clipping \(\widehat p_{a,m}(x)\) to \([0,1]\) is optional. Let \(a^*\) be a true best eligible program and let \(K_x=|\cA(x)|\).
Theorem 3 (Residual refinement frontier). Suppose that, conditional on the customer profiles, \(\{Y_{ia}:i=1,\ldots,n\}\) are independent for each \(a\), and the neighborhoods are fixed from information independent of these outcomes. For a prespecified nonempty level \(m\) and \(\alpha\in(0,1)\), with probability at least \(1-\alpha\),
If the same calibration outcomes select one of \(M\) levels, replace \(K_x\) by \(K_xM\).
Proof. Let \(\bar\rho_{a,m}=n_m^{-1}\sum_{i\in\cN_m}\rho_a(x_i)\). The conditional mean of (5) is \(\widetilde p_a(x)+\bar\rho_{a,m}\), and
Hoeffding’s inequality and a union bound over the \(K_x\) programs show that, simultaneously in \(a\),
Because \(\widehat a_m\) maximizes the fitted values,
A second union bound over the \(M\) levels proves the adaptive statement.□
The bound has an intuitive form. The mismatch \(\varepsilon_m^\rho\) decreases with refinement, while the sampling term grows as \(n_m\) falls. Let \(\widehat\varepsilon_m^\rho(x)\) be a validation estimate or upper-confidence estimate of the unknown mismatch. An estimated frontier chooses
The best level is not the most detailed one. It is the level at which a closer match is still worth the support it costs.
The true target residual is unknown at recommendation time. Let \(\widehat\rho(x)\) be a residual model trained separately from the target and reference outcomes. Write
Include level \(m=0\) with \(c_0(x)=0\). Define two observable discrepancies:
Convexity gives \(D_m^{\rm mean}\le D_m^{\rm max}\).
Corollary 4 (Operational peer-mean frontier). Suppose an independently constructed simultaneous band satisfies
Choose \(\widehat m(x)\) by minimizing either observable discrepancy, including level zero, without using \(p(x)\). Then, with probability at least \(1-\delta\),
and the selected program’s propensity regret is at most twice the corresponding right-hand side.
Proof. Use the decomposition
and apply the triangle inequality. Convexity gives the second inequality. The two-error argument in Theorem 3 gives the regret result.□
This is a latent-propensity guarantee only when the band covers \(\rho(x)\). Ordinary split conformal prediction instead gives a marginal statement about a future realized outcome. If the complete neighborhood algorithm is frozen before an independent calibration split, let \(\widehat p^{\rm op}(x)\) denote its final score vector and define
then the corrected split-conformal quantile \(q_{1-\alpha}\) satisfies
under exchangeability [4]. Realized-outcome regret is at most \(2q_{1-\alpha}\) on this event. This is marginal coverage, not pointwise coverage of the conditional mean.
Remark 5 (Known propensity coordinates). If domain knowledge fixes \(p_a(x)\) at zero or one, then its residual is known exactly and its uncertainty radius can be set to zero. Only the unknown coordinates need a residual band.
Corollary 6 (A confident recommendation). Assume \(K_x\ge2\). Let the corrected winner have empirical lead
On the event in Theorem 3, if \(\Delta_m(x)>2b_m(x)\), then \(\widehat a_m\) is the unique true best program.
Proof. Every true propensity is within \(b_m\) of its estimate. A fitted lead larger than \(2b_m\) stays positive even after both scores move by their largest permitted errors.□
Shrinkage replaces the raw correction by \(\lambda c_m\). Its uniform error is at most
Validation can choose \(\lambda\). In practice, rerank only when local evidence clears the relevant margin bound; otherwise retain the base choice or flag low confidence.
The mathematics maps to a short operating loop:
This retrieve-and-rank structure is common in large recommender systems [5]. Here the neighborhood and certificate decide whether local evidence is strong enough to modify the score.
KuaiRec contains nearly complete viewing records for 1,411 users and 3,327 videos [6]. We use the twelve largest video categories as program analogues. A user’s category propensity is the fraction of videos watched to completion. For each user, the first 60% of interactions form the observable history and the last 40% form future targets. In each repeated split, 60% of users estimate the models and out-of-fold residuals, 25% tune neighborhood choices, and 15% are untouched test users. A 32-coordinate SVD embedding is fit from estimation histories only.
Table 1. Mean chronological test performance over ten splits. Lower RMSE and regret are better; higher hit rate is better. Panel B uses ten fresh splits after the propensity-neighborhood rule was frozen.
| Method | RMSE | Top-1 regret | Hit rate |
|---|---|---|---|
| Panel A: behavioral representation | |||
| Static customer features | 0.126761 | 0.041357 | 34.46% |
| Static \(+\) behavioral embedding | 0.086310 | 0.038726 | 37.04% |
| Embedding-space shared neighbors | 0.085960 | 0.038530 | 37.09% |
| Embedding-space program-specific neighbors | 0.085564 | 0.039276 | 36.48% |
| Panel B: propensity-space confirmation | |||
| Static \(+\) behavioral embedding | 0.083850 | 0.036119 | 39.30% |
| Full-row propensity neighbors | 0.083498 | 0.036931 | 38.83% |
| Leave-one-program-out neighbors | 0.083833 | 0.037950 | 37.51% |
The embedding is the main result. Relative to static features, it lowers RMSE by 31.9%, lowers regret by 6.4%, and raises hit rate by 2.58 percentage points. RMSE improves in all ten splits, regret in eight, and hit rate in nine. The learned representation therefore captures useful future behavior before any local correction is applied.
Neighborhoods have a smaller role. Full-row propensity neighborhoods improve RMSE in nine of ten fresh splits, but their mean regret is 2.2% worse than direct embedding-based ranking. Leave-one-program-out matching is weaker. Better calibration does not necessarily identify a better winner.
A separate leakage-free operational experiment uses disjoint groups to fit the base model, fit \(\widehat\rho\), supply reference residuals, calibrate conformal scores, and evaluate the frozen rule. It aggregates 4,280 held-out-user evaluations.
Table 2. Operational fixed-radius diagnostic over twenty disjoint splits. Neighborhood use is the fraction of test users receiving a peer correction.
| Method | RMSE | Regret | Hit rate | Use |
|---|---|---|---|---|
| Base model | 0.110439 | 0.019361 | 49.18% | 0.00% |
| Theorem-faithful max frontier | 0.110438 | 0.019361 | 49.18% | 0.09% |
| Operational peer-mean frontier | 0.110129 | 0.019257 | 49.35% | 63.36% |
| Legacy score-space proxy | 0.110318 | 0.019429 | 49.00% | 100.00% |
The peer-mean frontier lowers RMSE by 0.000310; its descriptive paired 95% interval is \([-0.000560,-0.000060]\). Regret and hit-rate intervals include zero. End-to-end split conformal calibration attains 90.9% coverage at nominal 90%, but its mean regret bound, 0.5053, is slightly wider than the base benchmark, 0.5018. No top-two margin is wide enough to certify an individual winner. The experiments therefore support a modest calibration benefit, not an additional ranking gain.
The earlier fixed-radius pilot also displays the geometric mechanism: later blocks rapidly remove peers, and performance worsens when refinement outruns support. The support-aware rule avoids the deepest level. This is what the frontier is designed to control.
The practical recommendation is straightforward: use behavioral embeddings in the propensity model and rank its eligible outputs directly. Keep the neighborhood layer when local calibration, diagnostics, or confidence information is valuable. The theorem tells the system when a correction has enough support; it does not assume that useful local residual structure exists in every dataset.
The target also matters. Enrollment propensity is not the same as incremental enrollment, energy savings, or economic value. If a causal effect is identified under an appropriate assignment and policy-evaluation design, that effect can replace \(p_a(x)\) in the workflow [7]. Recent energy studies show the value of targeting heterogeneous conservation and retrofit effects [8, 9]. The present analysis is narrower: it studies enrollment propensity and the reliability of local correction.
The resulting story has three parts. Behavioral history supplies a useful geometry. Nested neighborhoods make local evidence more specific while spending peer support. The refinement frontier prices that trade and supports a calibrated recommendation, a retained base ranking, or an honest decision to abstain.
HTML edition · PDF version