# Diagnosis and Improvement Plan
## Diagnosis
### 1. The Model Has No Predictive Edge — Win Rates Are at Chance
The most critical finding is that all three markets sit at or below 50%: moneyline 50.0%, runline 46.6%, total 50.0%. This is not a calibration nuance — a model with genuine edge at the reported edge magnitudes (many comparisons show +0.10 to +0.30) should be winning at 55–60%+ across 58 games. The fact that it isn't means the `fair_prob` values being generated by the upstream pipeline are not predictive of actual outcomes. The market comparison layer (`_evaluate.py`) is functioning correctly as written, but it is faithfully amplifying noise from layers 1–5. The `edge` values reported are large and frequent (dozens of "green" picks per day), which itself is a red flag — genuine market inefficiencies of 10–30 percentage points don't exist at scale in MLB betting markets.
### 2. Total Market Calibration Is Severely Broken
The total market failures are the most instructive. Looking at the wrong predictions specifically flagged here: Game 1324 (actual 2-1, `over` bet, model edge +0.157), Game 1325 (actual 2-3, `over` bet, edge +0.178), Game 1331 (actual 3-1, `over` bet, edge +0.078), Game 1339 (actual 3-2, `over` bet, edge +0.121), Game 1347 (actual 4-3, `over` bet, edge +0.213), Game 1352 (actual 2-3, `over` bet, edge +0.128), Game 1353 (actual 4-0, `over` bet, edge +0.090), Game 1354 (actual 1-6, `over` bet, edge +0.077), Game 1367 (actual 3-2, `over` bet, edge +0.148), Game 1373 (actual 4-3, `over` bet, edge +0.289), Game 1375 (actual 1-3, `over` bet, edge +0.301). Every single one of these is a low-scoring game (combined scores of 3, 5, 4, 5, 7, 5, 4, 7, 5, 7, 4) that the model confidently predicted as overs. The projected totals are hidden (`proj_total=?`) which is itself a problem — but the pattern strongly suggests the model's PMF is systematically skewed toward higher run totals, likely because the Monte Carlo simulation in layer 5 is not regressing pitcher/lineup quality correctly, or the park factors (most are near 1.0, so that's not the driver) are being applied multiplicatively in a way that inflates expected run output.
### 3. The `proj_total=?` Gap Is a Critical Observability Failure
Every single game record shows `proj_total=?`. Looking at `_build_reasoning` in `_evaluate.py`, `projected_total` is only populated when `pred.market == "total"` and `pred.distribution` is not None. Given the total market is being evaluated (over/under verdicts are generated), either the distribution exists and the mean calculation is silently failing, or the reasoning is computed after the fact and `proj_total` is not being stored in a way the logging layer can read. Without knowing what the model is actually projecting for total runs, it is impossible to audit whether the fair_probs are reasonable. This is a root-cause observability gap.
### 4. Edge Threshold Is Too Permissive and the Verdict Function Ignores Sharp/RLM Signals
The `_verdict` function accepts `sharp` and `rlm` boolean arguments but does nothing with them — it only looks at `edge` vs `threshold`. With a 3% default threshold (`_DEFAULT_EDGE_THRESHOLD = 0.03`), almost everything becomes green. The data bears this out: the vast majority of comparisons are "green" (edge > 0.03), yet the win rate is 50%. Sharp money divergence and reverse line movement are computed but silently discarded in the verdict. Meanwhile, the model is generating enormous reported edges (+0.289 for Game 1373's over, which went under; +0.301 for Game 1375's over, which also went under) — edges this large against a liquid market like MLB are almost certainly model overconfidence, not genuine mispricing.
### 5. Runline Win Rate (46.6%) Suggests Directional Bias
The runline is the worst market at 46.6%, meaning the model is losing money even ignoring juice. Looking at the runline wrong picks: the model favors `away_plus` and `home_plus` (+1.5) frequently. Several of these involve games where the home team has `home_wp` between 0.52–0.58 (mild favorites), and the model still recommends taking the underdog +1.5 at high fair_prob (e.g., Game 1341: `away_plus` fair_prob=0.643 on a 1-14 blowout where home won 14-1 — wait, actual=1-14 means home scored 1, away scored 14, so that was actually a correct call). The systematic issue is that `home_plus` and `away_plus` predictions for the team that loses by more than 1.5 runs are failing at a high rate, suggesting the spread distribution from the Monte Carlo is too tight (underestimating blowout probability) or the runline consensus_implied is being mispriced due to alternate-line filtering logic in `_build_comparison`.
---
## Specific Improvement Suggestions
### 1. Raise the Edge Threshold and Make It Market-Specific
**File:** `services/model/src/mlb_model/market/_evaluate.py`
**Function:** `_edge_threshold()` and `_verdict()`
**Why:** A flat 3% threshold generates too many picks with no discriminatory power. Totals have the worst calibration and should require a higher threshold. Moneyline edges need to be larger to overcome juice. Market-specific thresholds will reduce pick volume and force the model to only act on its strongest signals.
```python
# In _evaluate.py, replace _edge_threshold() and _verdict() with:
_DEFAULT_EDGE_THRESHOLDS: dict[str, float] = {
"moneyline": 0.06, # Require 6pp edge on ML (juice ~4-5pp, need real signal)
"runline": 0.07, # Runline at 46.6% — be much more selective
"total": 0.08, # Totals badly miscalibrated, raise bar significantly
"f5_total": 0.08,
"nrfi": 0.06,
}
_DEFAULT_EDGE_THRESHOLD = 0.06 # fallback
def _edge_threshold(market: str | None = None) -> float:
"""Return per-market edge threshold, overridable via environment variable."""
raw = os.environ.get("EDGE_THRESHOLD_PCT", "")
if raw:
try:
return float(raw) / 100.0
except ValueError:
pass
if market is not None:
return _DEFAULT_EDGE_THRESHOLDS.get(market, _DEFAULT_EDGE_THRESHOLD)
return _DEFAULT_EDGE_THRESHOLD
def _verdict(edge: float, sharp: bool, rlm: bool, threshold: float) -> Verdict:
"""Assign verdict incorporating sharp money and RLM as modifiers."""
if edge < 0.0:
return Verdict.red
# Sharp divergence against our side is a veto even with positive edge
if sharp and edge < threshold * 1.5:
return Verdict.red
# RLM against our side downgrades green to yellow
if rlm and edge < threshold * 1.2:
return Verdict.yellow
if edge >= threshold:
return Verdict.green
return Verdict.yellow
```
Then update the call site in `_build_comparison` to pass the market:
```python
# In _build_comparison, replace:
threshold = _edge_threshold() # this is passed in from _evaluate already
# In _evaluate() -> for pred loop, replace:
threshold = _edge_threshold()
# with per-market threshold:
for pred in predictions:
market_threshold = _edge_threshold(pred.market)
comp = _build_comparison(
game_id=game_id,
model_run_id=model_run.id,
pred=pred,
odds_snapshots=odds_snapshots,
splits_snapshots=splits_snapshots,
threshold=market_threshold, # pass market-specific threshold
)
```
---
### 2. Fix `proj_total=?` — Ensure Projected Total Is Always Logged
**File:** `services/model/src/mlb_model/market/_evaluate.py`
**Function:** `_build_reasoning()`
**Why:** `proj_total=?` appears for every game, making it impossible to audit model calibration. The distribution mean calculation may be failing silently, or the `total/over` prediction may not exist for some games. Add a fallback that extracts projected total from `total/under` as well, add explicit error logging, and always surface the value.
```python
def _build_reasoning(
game_id: int,
model_run_id: int,
predictions: list[Prediction],
results: list[MarketComparison],
session: Session,
) -> dict:
from db.models import Game, ParkFactor, WeatherSnapshot
reasoning: dict = {}
for pred in predictions:
if pred.market in ("moneyline", "ml") and pred.side == "home" and pred.fair_prob is not None:
reasoning["home_win_prob"] = round(pred.fair_prob, 4)
# Extract projected total from either over or under distribution
if pred.market == "total" and pred.side in ("over", "under") and pred.distribution:
try:
items = [(float(k), v) for k, v in pred.distribution.items()]
if not items:
logger.warning(
"_build_reasoning: empty distribution game_id=%d pred_id=%s",
game_id, getattr(pred, "id", "?")
)
else:
mean_total = sum(k * v for k, v in items)
# Sanity check: MLB totals should be between 3 and 30
if 3.0 <= mean_total <= 30.0:
reasoning["projected_total"] = round(mean_total, 2)
else:
logger.warning(
"_build_reasoning: implausible projected_total=%.2f "
"game_id=%d — distribution may be malformed",
mean_total, game_id
)
reasoning["projected_total"] = round(mean_total, 2) # log anyway
except (ValueError, TypeError) as exc:
logger.warning(
"_build_reasoning: distribution parse error game_id=%d: %s",
game_id, exc
)
# Capture distribution percentiles for calibration auditing
if pred.market == "total" and pred.distribution and "projected_total" in reasoning:
try:
items_sorted = sorted((float(k), v) for k, v in pred.distribution.items())
cumulative = 0.0
p10 = p50 = p90 = None
for score, prob in items_sorted:
cumulative += prob
if p10 is None and cumulative >= 0.10:
p10 = score
if p50 is None and cumulative >= 0.50:
p50 = score
if p90 is None and cumulative >= 0.90:
p90 = score
reasoning["projected_total_percentiles"] = {
"p10": p10, "p50": p50, "p90": p90
}
except (ValueError, TypeError):
pass
reasoning["edges"] = [
{
"market": c.market,
"side": c.side,
"edge_pct": round(c.edge_pct, 4),
"verdict": c.verdict.value,
"sharp": c.sharp_divergence,
"rlm": c.reverse_line_movement,
}
for c in sorted(results, key=lambda c: abs(c.edge_pct), reverse=True)
]
reasoning["signals"] = {
"sharp_markets": [c.market for c in results if c.sharp_divergence],
"rlm_markets": [c.market for c in results if c.reverse_line_movement],
}
# ... rest of context building unchanged
```
---
### 3. Add a Pick Volume Limiter — Cap Simultaneous Green Picks Per Game
**File:** `services/model/src/mlb_model/market/_evaluate.py`
**Function:** `_evaluate()`
**Why:** Some games produce 3–4 simultaneous green picks (e.g., Game 1373: moneyline/home, runline/home_minus, total/over all green). In a correlated game, these are not independent bets — they share the same model error. When the model is wrong about a game, it's wrong about all of it at once. Limiting to the single highest-edge pick per game reduces correlated exposure and forces selectivity.
```python
_MAX_GREEN_PICKS_PER_GAME = int(os.environ.get("MAX_GREEN_PICKS_PER_GAME", "2"))
def _evaluate(game_id: int, session: Session) -> list[MarketComparison]:
model_run = get_latest_model_run_for_game(session, game_id)
if model_run is None:
logger.warning("evaluate_game: no model_run for game_id=%d", game_id)
return []
predictions = get_predictions(session, model_run.id)
odds_snapshots = get_odds_snapshots_for_game(session, game_id)
splits_snapshots = get_latest_splits_for_game(session, game_id)
now = datetime.now(UTC)
results: list[MarketComparison] = []
for pred in predictions:
market_threshold = _edge_threshold(pred.market)
comp = _build_comparison(
game_id=game_id,
model_run_id=model_run.id,
pred=pred,
odds_snapshots=odds_snapshots,
splits_snapshots=splits_snapshots,
threshold=market_threshold,
)
if comp is None:
continue
results.append(comp)
# Enforce per-game green pick cap: keep only the top-N by edge magnitude.
# Non-green picks are kept as-is for logging/auditing purposes.
green_results = [r for r in results if r.verdict == Verdict.green]
non_green_results = [r for r in results if r.verdict != Verdict.green]
if len(green_results) > _MAX_GREEN_PICKS_PER_GAME:
# Prioritize: (1) markets with sharp divergence supporting our side,
# (2) highest edge. Demote excess greens to yellow.
green_results_sorted = sorted(
green_results,
key=lambda c: (c.sharp_divergence, c.edge_pct),
reverse=True,
)
kept_green = green_results_sorted[:_MAX_GREEN_PICKS_PER_GAME]
demoted = green_results_sorted[_MAX_GREEN_PICKS_PER_GAME:]
for comp in demoted:
# Rebuild with yellow verdict — create new dataclass instance
comp = MarketComparison(
game_id=comp.game_id,
model_run_id=comp.model_run_id,
market=comp.market,
side=comp.side,
fair_prob=comp.fair_prob,
fair_price_american=comp.fair_price_american,
consensus_price_american=comp.consensus_price_american,
consensus_implied_prob=comp.consensus_implied_prob,
edge_pct=comp.edge_pct,
sharp_divergence=comp.sharp_divergence,
reverse_line_movement=comp.reverse_line_movement,
verdict=Verdict.yellow, # demoted due to