"""Scores v5 (2026-10-18; PENDING-WORK P-202, Matt: "Rebuild as you have described, and use three presidential elections as the
standard to set that constant"). Deterministic arithmetic, no fitted model. v5 = v4 with (1) the Partisan EAI redefined as the
neutral-year expectation (relative lean + Maine's baseline), (2) the U.S. House reference repaired so every unit has the same
zero, (3) the registration tile ("Partisan Edge") taken from the latest Secretary of State file, (4) the Performance Gap renamed
Hidden Lean, (5) a 2024 turnout figure.

Run in this folder after v5_prep.py:  SCORES_REG=<registration v7.2 out/> SCORES_RES=<results v3.14 folder> \
    SCORES_VF=<model-data/voterfile/aggregates/statewide_turnout_by_town_party.csv> SCORES_REGIONS=<geography/regions.csv> \
    SCORES_OUT=<output folder> python3 v5_scores.py

What is computed (plan: claude/scores-v4-plan-2026-10-17.md; audit: claude/scores-methodology-audit-2026-10-17.md; v5 decisions
in the 2026-10-18 scores chat, PENDING-WORK P-202):
  Raw vote margin    two-party margin (D minus R, share of the D+R vote) of the qualifying top-ticket general-election races
                     (President, Governor, U.S. Senate, U.S. House; first-choice votes; a race qualifies when both major parties
                     ran and non-major candidates took at most 15%; declared write-ins never count as the nominee), combined as
                     cycle values weighted 70% (2024) / 30% (2022). Inside a cycle the U.S. House race counts one third and the
                     statewide races share the other two thirds equally (a cycle with no House race: the statewide races equally;
                     a cycle with only a House race: the House race). Positive = Democratic. (This was the v4 "Partisan EAI".)
  Relative lean      the same races and weights, each race measured against Maine's result in that race. The U.S. House race is
                     measured against its district's result (which removes the House candidates' district-wide pull) and the
                     district's own lean against Maine in that cycle's statewide races is added back, so every unit's zero is
                     Maine. (v4's "Natural Lean" lacked the add-back and gave the two districts different zeros.)
  Maine baseline     Maine's two-party presidential margin minus the nation's, averaged over the last three presidential elections
                     (2016, 2020, 2024; the nation's totals are the FEC's official counts, below). The estimate of how Maine votes in
                     a neutral national year with the same candidates on every ballot. Re-estimated after each presidential election.
  Partisan EAI       = relative lean + Maine baseline: the unit's expected two-party margin in a neutral national year with average
                     Maine candidates. Maine's own Partisan EAI is the baseline. Rankings are those of the relative lean.
  Partisan Edge      enrolled Democrats minus Republicans as a share of enrolled D + R, ACTIVE voters, the LATEST SOS file
                     (the registration section of the place pages uses the same file); unenrolled share of active voters; inactive
                     share of the rolls (inactive / (active + inactive)). The Nov 5, 2024 file is kept for the Hidden Lean regression.
  Hidden Lean        raw vote margin minus the margin expected from registration: a vote-weighted statewide regression of town raw
                     vote on the town's active Democratic share and Republican share (Nov 2024, the file in force at the vote),
                     fitted on towns, applied at every level. (v4's "Performance Gap"; the number is unchanged.)
  Turnout 2024       voters in the state voter file who cast a ballot in Nov 2024, as a share of active registrants (Nov 2024).
  Electorate         who voted in Nov 2024 by registered party (state voter file aggregates; town totals only, no names).
  Competitive outlook  rules on the Partisan EAI, the window's results (each race's margin adjusted the same way) and turnout
                     headroom (non-voting partisans), see campaign_class(); office profile, drift (relative lean 2024 cycle minus
                     2018 cycle), counts, size class.
  Implied unenrolled lean (data and Methodology only, not a tile): 2024 President; partisans who voted assumed loyal at a stated
                     statewide rate (93%, range 90-97%), the rest of the two-party vote attributed to unenrolled voters.
  History 1990-2024  cycle values of the raw vote, relative lean, Maine baseline (the three presidential elections up to that
                     year), Partisan EAI, registration margin (all registered voters in the November file; active + inactive from
                     2008, "registered" before 2007) and Hidden Lean; rolling 70/30 values.
Levels: towns (registration units), counties, regions, state, and the current congressional, State Senate, State House and
county commissioner districts (towns split between districts are divided by their precincts' Nov 2024 registration: votes
by the precinct share of DEM + half UNE + half minor for the Democratic vote and REP + half UNE + half minor for the
Republican vote; registration by the precinct share of each group). Precincts: Nov 2024 active registration margin only.
Mapping of registration units to result lines, and the votes of a result line shared by several units (divided by their
registration), as in scores v2.3-v3.1 (scoring.py build_map, unchanged, credit: v2.3 audit fixes S1).
"""
import os, numpy as np, pandas as pd
os.environ.setdefault('SCORES_REG', '/home/claude/audit/reg/out')
os.environ.setdefault('SCORES_RES', '/home/claude/audit/er')
os.environ.setdefault('SCORES_GEN', 'gen_1990_2024.pkl')
NOVS = {2008: '2008-11-04', 2010: '2010-11-02', 2012: '2012-11-06', 2014: '2014-11-04', 2016: '2016-11-08',
        2018: '2018-11-06', 2020: '2020-11-03', 2022: '2022-11-08', 2024: '2024-11-05'}
os.environ['SCORES_INACTIVE'] = ','.join(NOVS.values())          # history rows 2008-2024: active + inactive (one rule, as v2.2-v3.1)
exec(open('scoring.py').read())                                  # race_table, race_votes, build_map, snapshot_for, rw, units, uname, ucty
VF = os.environ.get('SCORES_VF', '/mnt/user-data/uploads/Maine Political Data Project/model-data/voterfile/aggregates/statewide_turnout_by_town_party.csv')
REGIONS_CSV = os.environ.get('SCORES_REGIONS', '/mnt/user-data/uploads/Maine Political Data Project/geography/regions.csv')
OUT = os.environ.get('SCORES_OUT', '/home/claude/v4/out'); os.makedirs(OUT, exist_ok=True)
CHK = OUT + '/checks'; os.makedirs(CHK, exist_ok=True)
W = {2024: 0.7, 2022: 0.3}                                       # cycle weights (grid search of 492 schemes, audit b5)
HOUSE_W = 1 / 3                                                  # the U.S. House race's share of its cycle
SNAP = NOVS[2024]                                                # the registration file in force at the 2024 vote (Hidden Lean, district weights for votes)
BASE_N = 3                                                       # presidential elections in the Maine baseline (Matt, 2026-10-18)
US_PRES = {1980: (35480115, 43903230), 1984: (37577352, 54455472), 1988: (41809074, 48886597), 1992: (44909806, 39104550),
           1996: (47401185, 39197469), 2000: (50999897, 50456002), 2004: (59028444, 62040610), 2008: (69498516, 59948323),
           2012: (65915795, 60933504), 2016: (65853514, 62984828), 2020: (81283501, 74223975), 2024: (75019230, 77303568)}
# national (Democratic, Republican) presidential votes, FEC "Federal Elections" official results (two-party margin only is used)
LOYALTY, LOY_LO, LOY_HI = 0.93, 0.90, 0.97
CN = {'AND': 'Androscoggin', 'ARO': 'Aroostook', 'CUM': 'Cumberland', 'FRA': 'Franklin', 'HAN': 'Hancock', 'KEN': 'Kennebec', 'KNO': 'Knox', 'LIN': 'Lincoln',
      'OXF': 'Oxford', 'PEN': 'Penobscot', 'PIS': 'Piscataquis', 'SAG': 'Sagadahoc', 'SOM': 'Somerset', 'WAL': 'Waldo', 'WAS': 'Washington', 'YOR': 'York'}
G = ['DEM', 'REP', 'UNE', 'MIN']
for y, s in NOVS.items(): assert snapshot_for(y)[0] == s, (y, snapshot_for(y))

def iround(x):
    return pd.Series(np.floor(np.asarray(x, dtype='float64') + 0.5)).astype('Int64').values
def r1(x):
    x = pd.Series(np.asarray(x, dtype='float64'))
    return (np.sign(x) * np.floor(np.abs(x) * 10 + 0.5 + 1e-9) / 10).where(x.notna()) + 0.0
def lab(v):
    if pd.isna(v): return ''
    n = int(np.floor(abs(v) + 0.5 + 1e-9))
    return 'EVEN' if n == 0 else ('D+' if v > 0 else 'R+') + str(n)
def margin(d, r):
    s = np.asarray(d) + np.asarray(r)
    return np.where(s > 0, 100 * (np.asarray(d) - np.asarray(r)) / np.where(s > 0, s, 1), np.nan)

# ------------------------------------------------------------------ 1. races and unit x race votes
races = race_table()
races.to_csv(f'{OUT}/races.csv', index=False)
STATE_REF = {}                                                   # contest_id -> statewide (House: district) two-party margin, every result line
for cid in races[races.qualifies].contest_id:
    V = race_votes(cid); STATE_REF[cid] = float(margin(V.D.sum(), V.R.sum()))
rows = []
for y in sorted(races.year.unique()):
    snap, _ = snapshot_for(y)
    R = rw[rw.snapshot_date == snap].set_index('unit_id')[G + ['total']]
    R = R[R[G].sum(axis=1) > 0]
    for _, rc in races[(races.year == y) & races.qualifies].iterrows():
        V = race_votes(rc.contest_id)
        mp, dropped = build_map(y, R.index, V.index)
        Ru = R.loc[list(mp)].copy(); Ru['vu'] = [mp[u] for u in mp]
        Ru = Ru[Ru.vu.isin(V.index)]
        Ru['reg'] = Ru[G].sum(axis=1)
        sh = Ru.reg / Ru.groupby('vu').reg.transform('sum')     # a shared result line is divided by registration
        Ru['D'] = Ru.vu.map(V.D) * sh; Ru['R'] = Ru.vu.map(V.R) * sh
        Ru['shared'] = Ru.groupby('vu').vu.transform('size') > 1
        Ru['contest_id'] = rc.contest_id; Ru['year'] = y; Ru['office'] = rc.office; Ru['district'] = rc.district
        Ru['ref'] = STATE_REF[rc.contest_id]
        rows.append(Ru.reset_index().rename(columns={'index': 'unit_id'}))
RU = pd.concat(rows, ignore_index=True)
RU['county'] = RU.unit_id.map(ucty); RU['state'] = 'ME'
REGIONS = pd.read_csv(REGIONS_CSV); REGIONS = REGIONS[REGIONS.unit_id.notna()]
RU['region'] = RU.unit_id.map(dict(zip(REGIONS.unit_id, REGIONS.region_id)))
# Maine baseline: Maine minus nation, two-party presidential margin, mean of the last BASE_N presidential elections
PM = pd.read_csv('president_maine_1980_2024.csv').set_index('year')
PM['us_dem'] = [US_PRES[y][0] for y in PM.index]; PM['us_rep'] = [US_PRES[y][1] for y in PM.index]
PM['us_two_party_margin'] = 100 * (PM.us_dem - PM.us_rep) / (PM.us_dem + PM.us_rep)
PM['maine_minus_us'] = PM.maine_two_party_margin - PM.us_two_party_margin
def baseline_for(year):
    """mean of maine_minus_us over the last BASE_N presidential elections held on or before `year`"""
    ys = [y for y in PM.index if y <= year][-BASE_N:]
    return float(PM.loc[ys, 'maine_minus_us'].mean()) if len(ys) == BASE_N else np.nan
BASELINE = baseline_for(2024)
bt = PM.copy(); bt['baseline_last_%d' % BASE_N] = [baseline_for(y) for y in bt.index]
for n in (2, 4, 5, 7): bt[f'alt_mean_last_{n}'] = [float(PM.loc[[yy for yy in PM.index if yy <= y][-n:], 'maine_minus_us'].mean()) if len([yy for yy in PM.index if yy <= y]) >= n else np.nan for y in bt.index]
bt.round(3).to_csv(f'{CHK}/maine_baseline.csv')
_r7 = PM.loc[PM.index >= 2000, 'maine_minus_us']
open(f'{CHK}/maine_baseline.txt', 'w').write(
    f'Maine baseline = mean of (Maine two-party presidential margin - national two-party presidential margin) over the last {BASE_N} presidential elections '
    f'({", ".join(str(y) for y in PM.index[-BASE_N:])}): {BASELINE:+.2f} points.\n'
    f'Series since 2000: ' + ', '.join(f'{y} {v:+.1f}' for y, v in _r7.items()) + f'; sd {_r7.std():.2f}; standard error of a {BASE_N}-election mean {_r7.std() / BASE_N ** 0.5:.2f}.\n'
    f'Alternatives: last 2 {_r7.iloc[-2:].mean():+.2f}, last 4 {_r7.iloc[-4:].mean():+.2f}, last 5 {_r7.iloc[-5:].mean():+.2f}, all 7 since 2000 {_r7.mean():+.2f}.\n'
    f'Maine totals: every town line plus the UOCAVA lines (= the official statewide totals); national totals: FEC official results.\n')
print(open(f'{CHK}/maine_baseline.txt').read())

# U.S. House reference repair: the district's own lean against Maine in the cycle's statewide races is added back, so the House
# component is measured against Maine like the statewide races while the House candidates' district-wide pull is still removed.
DIST_LEAN = {}                                                   # (year, district) -> district lean vs Maine in that year's statewide races
for (y, d), g in RU[RU.office == 'U.S. House'].groupby(['year', 'district']):
    units_d = set(g.unit_id)
    sw = RU[(RU.year == y) & (RU.office != 'U.S. House') & RU.unit_id.isin(units_d)]
    vals = []
    for cid, gc in sw.groupby('contest_id'):
        vals.append(float(margin(gc.D.sum(), gc.R.sum())) - STATE_REF[cid])
    if vals: DIST_LEAN[(y, d)] = float(np.mean(vals))
for (y, d) in list(RU[RU.office == 'U.S. House'].groupby(['year', 'district']).groups):
    if (y, d) not in DIST_LEAN:                                  # a House-only cycle (1992, 1998, 2010): the mean of the adjacent cycles
        nb = [DIST_LEAN[k] for k in ((y - 2, d), (y + 2, d)) if k in DIST_LEAN]
        DIST_LEAN[(y, d)] = float(np.mean(nb)) if nb else 0.0
hm = RU.office == 'U.S. House'
RU['dist_lean'] = 0.0
RU.loc[hm, 'dist_lean'] = [DIST_LEAN[(y, d)] for y, d in zip(RU.loc[hm, 'year'], RU.loc[hm, 'district'])]
RU['ref'] = RU.ref - RU.dist_lean                                # House rows: district result minus the district's lean = "what Maine would have done here"
pd.Series(DIST_LEAN).rename_axis(['year', 'district']).rename('district_lean_vs_maine').reset_index().round(2).to_csv(f'{CHK}/district_lean_addback.csv', index=False)
RU['refDR'] = RU.ref / 100 * (RU.D + RU.R)                      # D minus R expected if the unit matched Maine's result (House: district result less the district's lean)
RU['tp'] = RU.D + RU.R

# ------------------------------------------------------------------ 2. slots, cycles, index
def slots(df, keys):
    """unit x year x office: the two-party margin and the Natural Lean of the slot (House: one district race per unit normally;
    an aggregate spanning both districts gets the vote-weighted reference)."""
    s = df.groupby(keys + ['year', 'office'])[['D', 'R', 'refDR', 'tp', 'DEM', 'REP']].sum().reset_index()
    s['m'] = margin(s.D, s.R); s['nl'] = np.where(s.tp > 0, 100 * (s.D - s.R - s.refDR) / s.tp.where(s.tp > 0), np.nan)
    s['env'] = s.m - s.nl
    return s
def cycles(df, keys):
    """unit x year: House one third, statewide races two thirds (equal), as described in the module docstring."""
    s = slots(df, keys)
    s['h'] = (s.office == 'U.S. House').astype(int)
    out = []
    for k, g in s.groupby(keys + ['year']):
        hh = g[g.h == 1]; oo = g[g.h == 0]
        def comb(col):
            if len(hh) and len(oo): return (1 - HOUSE_W) * oo[col].mean() + HOUSE_W * hh[col].mean()
            return g[col].mean()
        d = dict(zip(keys + ['year'], k if isinstance(k, tuple) else (k,)))
        d.update(m=comb('m'), nl=comb('nl'), n_races=len(g), races=', '.join(sorted(g.office)), tp=g.tp.mean(), tp_pres=g[g.office == 'President'].tp.sum())
        out.append(d)
    return pd.DataFrame(out), s
def index(cyc, keys, w=W):
    c = cyc[cyc.year.isin(w)].copy(); c['w'] = c.year.map(w)
    g = c.groupby(keys)
    o = pd.DataFrame({'eai': g.apply(lambda x: np.average(x.m, weights=x.w)), 'nl': g.apply(lambda x: np.average(x.nl, weights=x.w)),
                      'n_cycles': g.year.nunique(), 'n_races': g.n_races.sum()})
    o['env'] = o.eai - o.nl
    o['neutral'] = o.nl + BASELINE
    return o.reset_index()

# ------------------------------------------------------------------ 3. registration: Nov 2024 (Hidden Lean, votes) and the latest file (Partisan Edge)
_act_all = pd.read_csv(f'{REG}/registration_town_wide.csv', low_memory=False); _ina_all = pd.read_csv(f'{REG}/inactive_registration_town_wide.csv', low_memory=False)
SNAP_LATEST = max(_act_all.snapshot_date)                        # the latest Secretary of State file on hand (the place pages' registration section uses it)
def regframe(snap, suffix):
    act = _act_all[_act_all.snapshot_date == snap].set_index('unit_id').copy(); ina = _ina_all[_ina_all.snapshot_date == snap].set_index('unit_id').copy()
    for df_ in (act, ina):
        for c in MAJOR + MINOR_COLS:
            df_[c] = df_[c].fillna(0) if c in df_ else 0.0
        df_['MIN'] = df_[MINOR_COLS].sum(axis=1); df_['active_total'] = df_[G].sum(axis=1)
    f = pd.DataFrame({'dem_active' + suffix: act.DEM, 'rep_active' + suffix: act.REP, 'une_active' + suffix: act.UNE, 'min_active' + suffix: act.MIN, 'active_voters' + suffix: act.active_total})
    f['inactive_voters' + suffix] = ina.active_total.reindex(f.index).fillna(0)
    return f
CURREG = regframe(SNAP, '').join(regframe(SNAP_LATEST, '_latest'), how='outer').fillna(0)
CURREG['state'] = 'ME'; CURREG['county'] = CURREG.index.map(ucty); CURREG['region'] = CURREG.index.map(dict(zip(REGIONS.unit_id, REGIONS.region_id)))
RCOLS0 = ['dem_active', 'rep_active', 'une_active', 'min_active', 'active_voters', 'inactive_voters']
RCOLS = RCOLS0 + [c + '_latest' for c in RCOLS0]
print('registration files: Hidden Lean / votes', SNAP, '; Partisan Edge', SNAP_LATEST)

# ------------------------------------------------------------------ 4. voter file: who voted (2022, 2024) and non-voting partisans
vf = pd.read_csv(VF); vf = vf[vf.unit_id.str.startswith('MPD')]
PMAP = {'Democrat': 'D', 'Republican': 'R', 'Unaffiliated': 'U', 'Green': 'O', 'Libertarian ("Liberal" in file)': 'O'}
vf['p'] = vf.party.map(PMAP)
VOTED = {}
for el, y in (('VH22G', 2022), ('VH24G', 2024)):
    x = vf[vf.election == el].groupby(['unit_id', 'p'])[['registrants_now', 'voted']].sum().unstack('p').fillna(0)
    t = pd.DataFrame({f'voted_{y}_{p}': x[('voted', p)] for p in 'DRUO'})
    t[f'voted_{y}'] = t.sum(axis=1)
    t[f'nonvoting_{y}_D'] = x[('registrants_now', 'D')] - x[('voted', 'D')]
    t[f'nonvoting_{y}_R'] = x[('registrants_now', 'R')] - x[('voted', 'R')]
    VOTED[y] = t
VOTED = VOTED[2022].join(VOTED[2024], how='outer').fillna(0)
VOTED['state'] = 'ME'; VOTED['county'] = VOTED.index.map(ucty); VOTED['region'] = VOTED.index.map(dict(zip(REGIONS.unit_id, REGIONS.region_id)))
VCOLS = [c for c in VOTED.columns if c.startswith(('voted_', 'nonvoting_'))]

# ------------------------------------------------------------------ 5. districts: precinct weights (Nov 2024, active + inactive)
pa = pd.read_csv(f'{REG}/registration_precinct.csv', low_memory=False); pa = pa[pa.snapshot_date == SNAP].copy()
pi_ = pd.read_csv(f'{REG}/inactive_registration_precinct.csv', low_memory=False); pi_ = pi_[pi_.snapshot_date == SNAP].copy()
cur = pd.concat([pa, pi_], ignore_index=True)
for c_ in ['DEM', 'REP', 'UNE'] + MINOR_COLS: cur[c_] = cur[c_].fillna(0) if c_ in cur else 0.0
cur['MIN'] = cur[MINOR_COLS].sum(axis=1)
cur.loc[(cur.unit_id == 'MPD-0148') & cur.county_comm_dist.isna(), 'county_comm_dist'] = 3      # blank in the Nov 2024 file (v2 rule)
assert cur.cong_dist.notna().all() and cur.senate_dist.notna().all() and cur.house_dist.notna().all()
cur['cd'] = cur.cong_dist.astype(int).astype(str); cur['sd'] = cur.senate_dist.astype(int).astype(str); cur['hd'] = cur.house_dist.astype(int).astype(str)
cur['ccd'] = cur.county + '-' + cur.county_comm_dist.fillna(-1).astype(int).astype(str)
Wg = cur.groupby(['unit_id', 'county', 'cd', 'sd', 'hd', 'ccd'])[G].sum().reset_index()
tot = Wg.groupby('unit_id')[G].transform('sum'); allr = Wg[G].sum(axis=1); allt = allr.groupby(Wg.unit_id).transform('sum')
for g in G: Wg['w_' + g] = np.where(tot[g] > 0, Wg[g] / tot[g].replace(0, np.nan), allr / allt)
Wg['dv'] = Wg.DEM + 0.5 * Wg.UNE + 0.5 * Wg.MIN; Wg['rv'] = Wg.REP + 0.5 * Wg.UNE + 0.5 * Wg.MIN
for c in ('dv', 'rv'): Wg['w_' + c] = Wg[c] / Wg.groupby('unit_id')[c].transform('sum')
CD_CURRENT = Wg.groupby('unit_id').cd.first()
split = Wg.groupby('unit_id').size()
# the latest file's precinct shares, for dividing the latest registration of split towns (same district assignments as Nov 2024)
_pl = pd.read_csv(f'{REG}/registration_precinct.csv', low_memory=False); _pl = _pl[_pl.snapshot_date == SNAP_LATEST].copy()
_pli = pd.read_csv(f'{REG}/inactive_registration_precinct.csv', low_memory=False); _pli = _pli[_pli.snapshot_date == SNAP_LATEST].copy()
curL = pd.concat([_pl, _pli], ignore_index=True)
for c_ in ['DEM', 'REP', 'UNE'] + MINOR_COLS: curL[c_] = curL[c_].fillna(0) if c_ in curL else 0.0
curL['MIN'] = curL[MINOR_COLS].sum(axis=1)
curL.loc[(curL.unit_id == 'MPD-0148') & curL.county_comm_dist.isna(), 'county_comm_dist'] = 3
curL['cd'] = curL.cong_dist.astype(int).astype(str); curL['sd'] = curL.senate_dist.astype(int).astype(str); curL['hd'] = curL.house_dist.astype(int).astype(str)
curL['ccd'] = curL.county + '-' + curL.county_comm_dist.fillna(-1).astype(int).astype(str)
WgL = curL.groupby(['unit_id', 'county', 'cd', 'sd', 'hd', 'ccd'])[G].sum().reset_index()
totL = WgL.groupby('unit_id')[G].transform('sum'); allrL = WgL[G].sum(axis=1); alltL = allrL.groupby(WgL.unit_id).transform('sum')
for g in G: WgL['wL_' + g] = np.where(totL[g] > 0, WgL[g] / totL[g].replace(0, np.nan), allrL / alltL)
Wg = Wg.merge(WgL[['unit_id', 'cd', 'sd', 'hd', 'ccd', 'wL_DEM', 'wL_REP', 'wL_UNE', 'wL_MIN']], on=['unit_id', 'cd', 'sd', 'hd', 'ccd'], how='left')
for g in G: Wg['wL_' + g] = Wg['wL_' + g].fillna(Wg['w_' + g])   # a precinct missing from the latest file keeps its Nov 2024 share
def split_votes(df):
    x = df.merge(Wg[['unit_id', 'cd', 'sd', 'hd', 'ccd', 'w_dv', 'w_rv', 'w_DEM', 'w_REP', 'w_UNE', 'w_MIN']], on='unit_id', how='inner')
    x['D'] = x.D * x.w_dv; x['R'] = x.R * x.w_rv; x['refDR'] = x.ref / 100 * (x.D + x.R); x['tp'] = x.D + x.R
    for c, wc in (('DEM', 'w_DEM'), ('REP', 'w_REP'), ('UNE', 'w_UNE'), ('MIN', 'w_MIN')): x[c] = x[c] * x[wc]
    return x
def split_reg(df, cols_by_group):
    x = df.reset_index().merge(Wg[['unit_id', 'cd', 'sd', 'hd', 'ccd', 'w_DEM', 'w_REP', 'w_UNE', 'w_MIN', 'wL_DEM', 'wL_REP', 'wL_UNE', 'wL_MIN']], on='unit_id', how='inner')
    for c, wc in cols_by_group.items(): x[c] = x[c] * x[wc]
    return x
RUc = split_votes(RU)
CURREGc = split_reg(CURREG, {'dem_active': 'w_DEM', 'rep_active': 'w_REP', 'une_active': 'w_UNE', 'min_active': 'w_MIN',
                             'dem_active_latest': 'wL_DEM', 'rep_active_latest': 'wL_REP', 'une_active_latest': 'wL_UNE', 'min_active_latest': 'wL_MIN'})
CURREGc['active_voters'] = CURREGc[['dem_active', 'rep_active', 'une_active', 'min_active']].sum(axis=1)
CURREGc['inactive_voters'] = CURREGc.inactive_voters * (CURREGc.w_DEM + CURREGc.w_REP + CURREGc.w_UNE + CURREGc.w_MIN) / 4
CURREGc['active_voters_latest'] = CURREGc[['dem_active_latest', 'rep_active_latest', 'une_active_latest', 'min_active_latest']].sum(axis=1)
CURREGc['inactive_voters_latest'] = CURREGc.inactive_voters_latest * (CURREGc.wL_DEM + CURREGc.wL_REP + CURREGc.wL_UNE + CURREGc.wL_MIN) / 4
VOTEDc = split_reg(VOTED, {**{c: 'w_DEM' for c in VCOLS if c.endswith('_D')}, **{c: 'w_REP' for c in VCOLS if c.endswith('_R')},
                           **{c: 'w_UNE' for c in VCOLS if c.endswith('_U')}, **{c: 'w_MIN' for c in VCOLS if c.endswith('_O')}})
for y in (2022, 2024): VOTEDc[f'voted_{y}'] = VOTEDc[[f'voted_{y}_{p}' for p in 'DRUO']].sum(axis=1)

# ------------------------------------------------------------------ 6. assemble one level
SIZE = [(0, 100, 'under 100 votes', 20.5), (100, 300, '100-300 votes', 9.6), (300, 1000, '300-1,000 votes', 5.4), (1000, 1e12, '1,000+ votes', 3.7)]
def size_class(v):
    if pd.isna(v): return '', np.nan
    for lo, hi, n, sd in SIZE:
        if lo <= v < hi: return n, sd
def campaign_class(nl, worst_d, best_d, hr22, hr24, m22, m24):
    """nl: Partisan EAI (neutral-year expectation); worst_d/best_d: the window's least/most Democratic race margin, adjusted the
    same way (relative to Maine + baseline); hr: turnout headroom in points
    (non-voting Democrats minus non-voting Republicans as a share of the two-party vote); m: the cycle margins."""
    if pd.isna(nl): return '', ''
    side = 'D' if nl >= 0 else 'R'; party = 'Democratic' if side == 'D' else 'Republican'; other = 'Republican' if side == 'D' else 'Democratic'
    within5 = (worst_d < 5) if side == 'D' else (best_d > -5)          # the other side came within 5 points in some window race
    won = (worst_d < 0) if side == 'D' else (best_d > 0)
    reach = (f" The {other} side has won here in this window." if won else f" The {other} side has come within 5 points here.") if within5 else ''
    if abs(nl) >= 15 and not within5:
        return f'safe_{side}', f'Safe {party}: a neutral year reads {lab(nl)} here and no race of 2022-2024, adjusted for the year, came within 5 points.'
    if abs(nl) >= 7:
        return f'lean_{side}', f'Leans {party}: a neutral year reads {lab(nl)} here.{reach}'
    # battleground: |Partisan EAI| under 7. Until 2026-10-19 this split into battleground_turnout (in BOTH 2022 and 2024 the trailing
    # side's non-voting partisans outnumbered its deficit) and battleground_persuasion; the turnout rule fired for no town and two
    # State House districts, so Matt merged the two (2026-10-19). The headroom figures stay in the panel ("Turnout headroom" row).
    return 'battleground', f'Battleground: a neutral year reads {lab(nl)} here.'
GAP_COEF = None
def assemble(RUx, REGx, VOTx, keys, idcols=None, rank_within=None):
    global GAP_COEF
    idcols = idcols or keys
    cyc, sl = cycles(RUx, keys)
    o = index(cyc, keys)
    reg = REGx.groupby(keys)[RCOLS].sum()
    o = o.join(reg, on=keys)
    o['registration_margin_2024'] = margin(o.dem_active, o.rep_active)                       # Nov 2024 file (the Hidden Lean basis)
    o['partisan_edge'] = margin(o.dem_active_latest, o.rep_active_latest)                    # latest file (the tile)
    o['unenrolled_share_pct'] = 100 * o.une_active_latest / o.active_voters_latest.replace(0, np.nan)
    o['inactive_share_pct'] = 100 * o.inactive_voters_latest / (o.active_voters_latest + o.inactive_voters_latest).replace(0, np.nan)
    # performance gap: fitted on towns (first call), applied everywhere
    o['s_dem'] = o.dem_active / o.active_voters.replace(0, np.nan); o['s_rep'] = o.rep_active / o.active_voters.replace(0, np.nan)
    tp24 = cyc[cyc.year == 2024].set_index(keys).tp_pres if len(keys) == 1 else cyc[cyc.year == 2024].set_index(keys).tp_pres
    o['two_party_votes_2024'] = o.set_index(keys).index.map(tp24).values if len(keys) == 1 else [tp24.get(tuple(k), np.nan) for k in o[keys].itertuples(index=False)]
    if GAP_COEF is None:
        ok = o.dropna(subset=['eai', 's_dem', 's_rep', 'two_party_votes_2024']); ok = ok[ok.two_party_votes_2024 > 0]
        X = np.column_stack([np.ones(len(ok)), ok.s_dem, ok.s_rep]); sw = np.sqrt(ok.two_party_votes_2024.values)
        GAP_COEF, *_ = np.linalg.lstsq(X * sw[:, None], ok.eai.values * sw, rcond=None)
        pred = X @ GAP_COEF; resid = ok.eai.values - pred
        r2 = 1 - np.average(resid ** 2, weights=ok.two_party_votes_2024) / np.average((ok.eai - np.average(ok.eai, weights=ok.two_party_votes_2024)) ** 2, weights=ok.two_party_votes_2024)
        open(f'{CHK}/hidden_lean_regression.txt', 'w').write(
            f'Hidden Lean regression (towns, weights = 2024 President two-party votes, n = {len(ok)}):\n'
            f'  expected raw vote margin = {GAP_COEF[0]:.2f} + {GAP_COEF[1]:.2f} x (active Democratic share) + {GAP_COEF[2]:.2f} x (active Republican share)\n'
            f'  shares as fractions of active voters, Nov 5, 2024. Weighted R2 = {r2:.3f}; residual sd (weighted) = {np.sqrt(np.average(resid**2, weights=ok.two_party_votes_2024)):.2f} points.\n')
    o['hidden_lean_expected'] = GAP_COEF[0] + GAP_COEF[1] * o.s_dem + GAP_COEF[2] * o.s_rep
    o['hidden_lean'] = o.eai - o.hidden_lean_expected
    o['turnout_2024_pct'] = np.nan
    # electorate and headroom
    vt = VOTx.groupby(keys)[VCOLS].sum(); o = o.join(vt, on=keys)
    for y in (2022, 2024):
        for p in 'DRUO': o[f'voters_{y}_{p}_pct'] = 100 * o[f'voted_{y}_{p}'] / o[f'voted_{y}'].replace(0, np.nan)
        tpy = cyc[cyc.year == y].set_index(keys).tp
        o[f'two_party_votes_{y}_mean'] = [tpy.get(k if len(keys) > 1 else k[0], np.nan) for k in o[keys].itertuples(index=False)]
        o[f'headroom_votes_{y}'] = o[f'nonvoting_{y}_D'] - o[f'nonvoting_{y}_R']
        o[f'turnout_headroom_{y}_pts'] = 100 * o[f'headroom_votes_{y}'] / o[f'two_party_votes_{y}_mean'].replace(0, np.nan)
    o['turnout_2024_pct'] = 100 * o.voted_2024 / o.active_voters.replace(0, np.nan)          # voter-file ballots / active registrants, Nov 2024
    # office profile and reach (the window's races)
    slw = sl[sl.year.isin(W)]   # the window's races only (cycles() ran on every year so that drift can use 2018)
    piv = slw.pivot_table(index=keys, columns=['office', 'year'], values='m')
    def col(off, y):
        return piv[(off, y)] if (off, y) in piv.columns else pd.Series(np.nan, index=piv.index)
    prof = pd.DataFrame({'margin_president_2024': col('President', 2024), 'margin_governor_2022': col('Governor', 2022), 'margin_senate_2024': col('U.S. Senate', 2024),
                         'margin_senate_2022': col('U.S. Senate', 2022), 'margin_house_2024': col('U.S. House', 2024), 'margin_house_2022': col('U.S. House', 2022)})
    prof['office_spread'] = prof.max(axis=1) - prof.min(axis=1)
    prof['best_dem_margin'] = slw.groupby(keys).m.max(); prof['worst_dem_margin'] = slw.groupby(keys).m.min()
    prof['best_dem_margin_adj'] = slw.groupby(keys).nl.max() + BASELINE; prof['worst_dem_margin_adj'] = slw.groupby(keys).nl.min() + BASELINE
    o = o.join(prof, on=keys)
    cy = cyc.set_index(keys + ['year'])
    for y in (2018, 2022, 2024):
        o[f'eai_cycle_{y}'] = [cy.m.get((*k, y) if len(keys) > 1 else (k[0], y), np.nan) for k in o[keys].itertuples(index=False)]
        o[f'natural_lean_cycle_{y}'] = [cy.nl.get((*k, y) if len(keys) > 1 else (k[0], y), np.nan) for k in o[keys].itertuples(index=False)]
    o['drift'] = o.natural_lean_cycle_2024 - o.natural_lean_cycle_2018          # relative lean, 2024 cycle minus 2018 cycle
    sc = [size_class(v) for v in o.two_party_votes_2024.fillna(o.two_party_votes_2022_mean)]   # a unit without a 2024 race: its 2022 size
    o['size_class'] = [s[0] for s in sc]; o['cycle_change_sd'] = [s[1] for s in sc]
    cc = [campaign_class(*t) for t in zip(o.neutral, o.worst_dem_margin_adj, o.best_dem_margin_adj, o.turnout_headroom_2022_pts, o.turnout_headroom_2024_pts, o.eai_cycle_2022, o.eai_cycle_2024)]
    o['campaign_class'] = [c[0] for c in cc]; o['campaign_reading'] = [c[1] for c in cc]
    # implied unenrolled lean, 2024 President (partisans who voted loyal at the stated rate)
    pres = RUx[(RUx.year == 2024) & (RUx.office == 'President')].groupby(keys)[['D', 'R']].sum()
    o = o.join(pres.rename(columns={'D': 'pres24_D', 'R': 'pres24_R'}), on=keys)
    for nm, L in (('implied_unenrolled_lean_2024', LOYALTY), ('implied_unenrolled_lean_2024_low', LOY_LO), ('implied_unenrolled_lean_2024_high', LOY_HI)):
        ud = o.pres24_D - L * o.voted_2024_D - (1 - L) * o.voted_2024_R; ur = o.pres24_R - L * o.voted_2024_R - (1 - L) * o.voted_2024_D
        v = np.where((ud + ur) >= 20, 100 * (ud - ur) / (ud + ur).replace(0, np.nan), np.nan)
        o[nm] = np.clip(v, -100, 100)
    o['implied_unenrolled_lean_range'] = o.implied_unenrolled_lean_2024_low.combine(o.implied_unenrolled_lean_2024_high, lambda a, b: '' if pd.isna(a) or pd.isna(b) else f'{lab(min(a, b))} to {lab(max(a, b))}')
    # names, rounding, labels, ranks
    out = o[idcols].copy()
    for src, nm in (('neutral', 'partisan_eai'), ('eai', 'raw_vote_margin'), ('nl', 'relative_lean'), ('env', 'statewide_environment'),
                    ('partisan_edge', 'partisan_edge'), ('registration_margin_2024', 'registration_margin_2024'),
                    ('hidden_lean', 'hidden_lean'), ('hidden_lean_expected', 'hidden_lean_expected')):
        out[nm] = r1(o[src].values).values
    out['maine_baseline'] = round(BASELINE, 1)
    for nm in ('partisan_eai', 'raw_vote_margin', 'relative_lean', 'partisan_edge', 'hidden_lean'): out[nm + '_label'] = out[nm].map(lab)
    out['partisan_edge_snapshot'] = SNAP_LATEST; out['registration_2024_snapshot'] = SNAP
    out['turnout_2024_pct'] = r1(o.turnout_2024_pct.values).values
    for c in ('unenrolled_share_pct', 'inactive_share_pct'): out[c] = r1(o[c].values).values
    for c in RCOLS: out[c] = iround(o[c].values)
    for c in ['voted_2024', 'voted_2022'] + [f'voted_{y}_{p}' for y in (2024, 2022) for p in 'DRUO']: out[c] = iround(o[c].fillna(0).values)
    for c in [f'voters_{y}_{p}_pct' for y in (2024, 2022) for p in 'DRUO'] + ['turnout_headroom_2024_pts', 'turnout_headroom_2022_pts']: out[c] = r1(o[c].values).values
    for c in ('headroom_votes_2024', 'headroom_votes_2022', 'two_party_votes_2024'): out[c] = iround(o[c].values)
    for c in ['margin_president_2024', 'margin_governor_2022', 'margin_senate_2024', 'margin_senate_2022', 'margin_house_2024', 'margin_house_2022', 'office_spread',
              'best_dem_margin', 'worst_dem_margin', 'best_dem_margin_adj', 'worst_dem_margin_adj', 'eai_cycle_2018', 'eai_cycle_2022', 'eai_cycle_2024', 'natural_lean_cycle_2018', 'natural_lean_cycle_2022', 'natural_lean_cycle_2024', 'drift',
              'implied_unenrolled_lean_2024', 'implied_unenrolled_lean_2024_low', 'implied_unenrolled_lean_2024_high']:
        out[c] = r1(o[c].values).values
    out['implied_unenrolled_lean_range'] = o.implied_unenrolled_lean_range.values
    out['size_class'] = o.size_class.values; out['cycle_change_sd'] = o.cycle_change_sd.values
    out['campaign_class'] = o.campaign_class.values; out['campaign_reading'] = o.campaign_reading.values
    for nm in ('partisan_eai', 'raw_vote_margin', 'partisan_edge', 'hidden_lean'):
        out[nm + '_rank_most_D'] = out[nm].rank(ascending=False, method='min').astype('Int64')
        out[nm + '_rank_most_R'] = out[nm].rank(ascending=True, method='min').astype('Int64')
        if rank_within: out[nm + f'_rank_in_{rank_within}_most_D'] = out[nm].groupby(out[rank_within]).rank(ascending=False, method='min').astype('Int64')
    out['n_cycles'] = o.n_cycles.values; out['n_races'] = o.n_races.values; out['races_in_window'] = o[keys].merge(
        cyc[cyc.year.isin(W)].groupby(keys).races.agg(lambda x: ' | '.join(f'{r}' for r in x)).reset_index(), on=keys, how='left').races.values
    out['units_in_ranking'] = len(out)
    return out.sort_values('partisan_eai', ascending=False, kind='mergesort').reset_index(drop=True), cyc, sl

T = {}
RUw = RU[RU.year.isin(W)]
tw, cyc_t, sl_t = assemble(RU, CURREG, VOTED, ['unit_id'], rank_within=None)
tw.insert(1, 'name', tw.unit_id.map(uname)); tw.insert(2, 'unit_type', tw.unit_id.map(dict(zip(units.unit_id, units.unit_type))))
tw.insert(3, 'county', tw.unit_id.map(ucty)); tw.insert(4, 'county_name', tw.county.map(CN))
for nm in ('partisan_eai', 'raw_vote_margin', 'partisan_edge', 'hidden_lean'):
    tw[nm + '_rank_in_county_most_D'] = tw[nm].groupby(tw.county).rank(ascending=False, method='min').astype('Int64')
tw['shared_results_line'] = tw.unit_id.map(RUw.groupby('unit_id').shared.any()).fillna(False).values
T['towns'] = tw
st, cyc_s, _ = assemble(RU, CURREG, VOTED, ['state']); st['campaign_class'] = ''; st['campaign_reading'] = ''; T['statewide'] = st   # Maine's relative lean is 0 by definition: its EAI is the baseline; no competitive outlook
assert abs(float(st.partisan_eai.iloc[0]) - round(BASELINE, 1)) <= 0.11 and abs(float(st.relative_lean.iloc[0])) <= 0.11, st[['partisan_eai', 'relative_lean']]
co, cyc_c, _ = assemble(RU, CURREG, VOTED, ['county']); co.insert(1, 'county_name', co.county.map(CN)); T['counties'] = co
rg, _, _ = assemble(RU.dropna(subset=['region']), CURREG.dropna(subset=['region']), VOTED.dropna(subset=['region']), ['region'])
rg = rg.rename(columns={'region': 'region_id'}); rg.insert(1, 'region', rg.region_id.map(dict(zip(REGIONS.region_id, REGIONS.region)))); T['regions'] = rg
def district(key, outname, idfix=int):
    d, _, _ = assemble(RUc, CURREGc, VOTEDc, [key])
    d['towns'] = d[key].map(Wg.groupby(key).unit_id.agg(lambda s_: '; '.join(sorted(uname[u] for u in set(s_)))))
    d['n_split_towns'] = d[key].map(Wg.groupby(key).unit_id.agg(lambda s_: sum(split[u] > 1 for u in set(s_))))
    d = d.rename(columns={key: outname}); d[outname] = d[outname].map(idfix)
    return d.sort_values('partisan_eai', ascending=False, kind='mergesort').reset_index(drop=True)
T['congressional_districts'] = district('cd', 'congressional_district')
T['state_senate_districts'] = district('sd', 'senate_district')
T['state_house_districts'] = district('hd', 'house_district')
cc = district('ccd', 'ccd', str); cc.insert(1, 'county', cc.ccd.str[:3]); cc.insert(2, 'county_name', cc.county.map(CN)); cc.insert(3, 'commissioner_district', cc.ccd.str[4:].astype(int))
for nm in ('partisan_eai', 'raw_vote_margin', 'partisan_edge', 'hidden_lean'):
    cc[nm + '_rank_in_county_most_D'] = cc[nm].groupby(cc.county).rank(ascending=False, method='min').astype('Int64')
T['county_commissioner_districts'] = cc.drop(columns=['ccd'])

# precincts: the latest file's active registration margin (the place pages' precinct table uses the same file)
pa = _pl.copy()
pa['MINp'] = pa[[c for c in MINOR_COLS if c in pa]].fillna(0).sum(axis=1)
pr = pa.groupby(['unit_id', 'ward_precinct'])[['DEM', 'REP', 'UNE', 'MINp']].sum().reset_index()
pr = pr[pr.unit_id.map(lambda u: (pa.unit_id == u).sum()) > 1]
pr['active_voters'] = pr[['DEM', 'REP', 'UNE', 'MINp']].sum(axis=1).astype(int)
pr['partisan_edge'] = r1(margin(pr.DEM, pr.REP)).values; pr['partisan_edge_label'] = pr.partisan_edge.map(lab)
pr['unenrolled_share_pct'] = r1(100 * pr.UNE / pr.active_voters.replace(0, np.nan)).values
pr['name'] = pr.unit_id.map(uname); pr['snapshot'] = SNAP_LATEST
T['precincts'] = pr.rename(columns={'DEM': 'dem_active', 'REP': 'rep_active', 'UNE': 'une_active', 'MINp': 'min_active'})[
    ['unit_id', 'name', 'ward_precinct', 'snapshot', 'dem_active', 'rep_active', 'une_active', 'min_active', 'active_voters', 'partisan_edge', 'partisan_edge_label', 'unenrolled_share_pct']]

# ------------------------------------------------------------------ 7. history 1990-2024
def regmargin_hist(keys, reg_split=None):
    out = []
    for y in sorted(races.year.unique()):
        snap, _ = snapshot_for(y)
        R = rw[rw.snapshot_date == snap].set_index('unit_id')[G]
        R = R[R.sum(axis=1) > 0].copy(); R['state'] = 'ME'; R['county'] = R.index.map(ucty); R['region'] = R.index.map(dict(zip(REGIONS.unit_id, REGIONS.region_id))); R['cd_current'] = R.index.map(CD_CURRENT)
        if reg_split is not None: R = split_reg(R, {'DEM': 'w_DEM', 'REP': 'w_REP', 'UNE': 'w_UNE', 'MIN': 'w_MIN'})
        g = R.groupby(keys)[G].sum(); g['year'] = y; g['reg_margin'] = margin(g.DEM, g.REP); g['s_dem'] = g.DEM / g[G].sum(axis=1); g['s_rep'] = g.REP / g[G].sum(axis=1)
        g['snapshot'] = snap
        out.append(g.reset_index())
    return pd.concat(out, ignore_index=True)
HIST_COEF = {}
def history(RUx, keys, reg_split=None, fit=False):
    cyc, _ = cycles(RUx, keys)
    rm = regmargin_hist(keys, reg_split)
    h = cyc.merge(rm[keys + ['year', 'reg_margin', 's_dem', 's_rep', 'snapshot']], on=keys + ['year'], how='left')
    h = h.sort_values(keys + ['year']).reset_index(drop=True)
    # rolling 70/30 with the previous cycle (= the tile for 2024)
    prev = h.groupby(keys)[['m', 'nl', 'year']].shift(1)
    okp = (h.year - prev.year == 2)
    h['eai_rolling'] = np.where(okp, 0.7 * h.m + 0.3 * prev.m, np.nan); h['nl_rolling'] = np.where(okp, 0.7 * h.nl + 0.3 * prev.nl, np.nan)
    # gap per cycle: towns fit the regression each cycle (weights = two-party votes); other levels apply the town coefficients
    h['gap'] = np.nan
    for y, g in h.groupby('year'):
        ok = g.dropna(subset=['m', 's_dem', 's_rep']); ok = ok[ok.tp > 0]
        if fit:
            X = np.column_stack([np.ones(len(ok)), ok.s_dem, ok.s_rep]); sw = np.sqrt(ok.tp.values)
            HIST_COEF[y], *_ = np.linalg.lstsq(X * sw[:, None], ok.m.values * sw, rcond=None)
        c = HIST_COEF.get(y)
        if c is not None: h.loc[ok.index, 'gap'] = ok.m - (c[0] + c[1] * ok.s_dem + c[2] * ok.s_rep)
    o = h[keys + ['year', 'n_races', 'races']].copy()
    bt_ = h.year.map(baseline_for)
    o['raw_vote_cycle'] = r1(h.m.values).values; o['relative_lean_cycle'] = r1(h.nl.values).values; o['maine_baseline'] = r1(bt_.values).values
    o['partisan_eai_cycle'] = r1((h.nl + bt_).values).values; o['registration_margin_cycle'] = r1(h.reg_margin.values).values
    o['hidden_lean_cycle'] = r1(h.gap.values).values; o['raw_vote_rolling_70_30'] = r1(h.eai_rolling.values).values; o['relative_lean_rolling_70_30'] = r1(h.nl_rolling.values).values
    o['partisan_eai_rolling_70_30'] = r1((h.nl_rolling + bt_).values).values
    o['two_party_votes_mean'] = iround(h.tp.values); o['registration_snapshot'] = h.snapshot.values
    o['registration_basis'] = np.where(h.year >= 2008, 'active + inactive voters, November file', 'registered voters (status not stated before 2007; the latest file of the year on hand, see registration_snapshot)')
    return o
H = {}
H['towns'] = history(RU, ['unit_id'], fit=True); H['towns'].insert(1, 'name', H['towns'].unit_id.map(uname)); H['towns'].insert(2, 'county', H['towns'].unit_id.map(ucty))
H['counties'] = history(RU, ['county'])
hs = history(RU, ['state']).rename(columns={'state': 'unit'})
RU['cd_current'] = RU.unit_id.map(CD_CURRENT)
hc = history(RU.dropna(subset=['cd_current']), ['cd_current']).rename(columns={'cd_current': 'unit'}); hc['unit'] = 'CD' + hc.unit.astype(str)
H['cd_state'] = pd.concat([hs, hc], ignore_index=True)
for y in (2024,):   # the 2024 rolling value is the tile
    t = T['towns'].set_index('unit_id').partisan_eai; m24 = H['towns'].year == 2024
    d = (H['towns'].loc[m24, 'partisan_eai_rolling_70_30'] - H['towns'].loc[m24, 'unit_id'].map(t)).abs()
    assert d.fillna(0).max() <= 0.1001, d.max()
open(f'{CHK}/history_gap_coefficients.csv', 'w').write('year,intercept,dem_share,rep_share\n' + '\n'.join(f'{y},{c[0]:.3f},{c[1]:.3f},{c[2]:.3f}' for y, c in sorted(HIST_COEF.items())) + '\n')

# ------------------------------------------------------------------ 8. write
for k, v in T.items(): v.to_csv(f'{OUT}/scores_{k}.csv', index=False)
for k, v in H.items(): v.to_csv(f'{OUT}/scores_history_{k}.csv', index=False)
pd.to_pickle(dict(T=T, H=H, RU=RU, RUc=RUc, Wg=Wg, GAP_COEF=GAP_COEF, HIST_COEF=HIST_COEF, cyc_t=cyc_t, sl_t=sl_t, STATE_REF=STATE_REF, races=races), f'{OUT}/v5_state.pkl')
print({k: len(v) for k, v in T.items()}, {k: len(v) for k, v in H.items()})
print(open(f'{CHK}/hidden_lean_regression.txt').read())
print(T['statewide'].T.to_string())
print(T['towns'].campaign_class.value_counts())