# Maine polling history — dataset v0.4.7 (2026-10-29)

**v0.4.7 (2026-10-29, scheduled freshness check C — polls)** adds one Maine poll through `additions/`
(`make_additions_2026-10-29.py` names the source of every figure; patch `build/polling_patch_2026-10-29.zip`), read from the
pollster's own document: **CNN poll conducted by SSRS, Oct 20–26 2026** ("Maine State Poll 2", embargoed to Oct 29 5:00 a.m.;
33 pages, topline pp. 1–8 and crosstabs; posted by CNN on DocumentCloud, document 28856725 — read through DocumentCloud's text
layer in the iMac built-in browser, as DocumentCloud refuses the cloud workspace, so no PDF copy is in `sources/`; its address
is in `polls.csv`). 783 registered voters (L2 voter-list sample 722 + SSRS Opinion Panel 61), online and telephone; likely-voter
results weight all 783 interviews by modelled turnout (no separate LV n), LV MoE ±4.5. Likely voters, already-voted merged with
vote-today and a lean follow-up (flag "includes leaners"): U.S. Senate Jackson 49 / Collins 45 / neither 4 / other 2; Governor
Pingree 53 / Charles 39 / neither 4 / other 4 — a two-way question (Rick Bennett not offered), as in the Aug 31–Sept 6 CNN wave
(P2026-0906-ssrs-cnn), whose class, sponsor and type are copied. "No opinion" under 0.5 not recorded. Grade A; party crosstabs
not added. 688 polls (+1), 2,158 questions (+2), 7,386 results (+8). `poll_question_links.csv` got two rows ("not yet held").
Rebuild: as for v0.4.6, then unzip `build/polling_patch_2026-10-29.zip` (or, from v0.4.6 `data/`, just
`python3 scripts/apply_additions.py`, then add the two link rows). Not updated: `pollsters.csv` counts.


**v0.4.6 (2026-10-28, scheduled freshness check C — polls)** adds one Maine poll through `additions/`
(`make_additions_2026-10-28.py` names the source of every figure; patch `build/polling_patch_2026-10-28.zip`), read from the
pollster's own documents (copies in `sources/added_2026-10-28/`): **Echelon Insights for NetChoice, Oct 16–23 2026** (the
"Fall 2026 Antitrust Tech Survey", released Oct 27: a national likely-voter survey, base 1,004, with a Maine oversample of
**413 likely voters, MoE ±6.3**; online panels matched to the L2 voter file plus text-to-web; 28-page topline and 7-page
Echelon release memo). Maine-only items: U.S. Senate Jackson 51 / Collins 44 / someone else 1 / unsure 4; Governor Pingree 51
/ Charles 38 / Bennett 7 / did not vote for this office 1 / unsure 3; generic U.S. House ballot (ME column, definitely +
probably + lean) D 54 / R 42 / unsure 4. Already-voted choices (21% of the Maine sample) are merged with vote-today and a
leaner follow-up, so flag "includes leaners". "*" (under 0.5%) cells not recorded, as for Fox (v0.4.3). Pollster class,
poll type and partisan flag copied from the April 2026 Echelon/NetChoice poll (P2026-0409-echelon-insights) so the two are
treated alike; sponsor NetChoice. Grade B (topline, no crosstabs). 687 polls (+1), 2,156 questions (+3), 7,378 results (+12).
`poll_question_links.csv` got three rows (Senate and Governor "not yet held"; the generic ballot "generic party ballot for
both House districts combined"). Rebuild: as for v0.4.5, then unzip `build/polling_patch_2026-10-28.zip` (or, from v0.4.5
`data/`, just `python3 scripts/apply_additions.py`, then add the three link rows). Not updated: `pollsters.csv` counts.

**v0.4.5 (2026-10-27, scheduled freshness check C — polls)** adds one Maine poll through `additions/`
(`make_additions_2026-10-27.py` names the source of every figure; patch `build/polling_patch_2026-10-27.zip`), read from the
pollster's own document (copy in `sources/added_2026-10-27/`): **Marist Poll, Oct 22–25 2026** ("The Midterm Elections in Maine,
October 2026", released Oct 27; 86-page table packet; 1,449 registered voters ±4.1, phone/text/online; likely voters ±6.6 — a
likely-voter weight applied to all 1,449 interviews, no separate LV n printed). Recorded, as Marist headlines them, the
likely-voter tables: U.S. Senate Jackson 49 / Collins 47 / another 1 / undecided 3; Governor Pingree 44 / Charles 44 / Bennett 6 /
another 1 / undecided 6; generic U.S. House ballot D 52 / R 47; plus the CD1 and CD2 "Region" columns of each (which, as in the
Oct 1–4 wave, exclude Kennebec County). Registered-voter toplines (Senate 53 / 44, Governor 51 / 40 / 7, generic 55 / 44) are in
each question's `context`. Pollster name, class, type and the "includes leaners" flag copied from the Oct 1–4 Marist wave
(P2026-1004-marist-college) so the two are treated alike; "<1%" stored as 0.5 as there. Grade A. Party crosstabs not added.
686 polls (+1), 2,153 questions (+9), 7,366 results (+48). All 2026 questions: `poll_question_links.csv` got nine rows ("not yet
held"; the statewide generic ballot "generic party ballot for both House districts combined"), so no link or accuracy row
changed. Rebuild: as for v0.4.4, then unzip `build/polling_patch_2026-10-27.zip` (or, from v0.4.4 `data/`, just
`python3 scripts/apply_additions.py`, then add the nine link rows). Not updated: `pollsters.csv` counts.

**v0.4.4 (2026-10-26, scheduled freshness check C — polls)** adds one Maine poll through `additions/`
(`make_additions_2026-10-26.py` names the source of every figure; patch `build/polling_patch_2026-10-26.zip`), read from the
pollster's own document (copy in `sources/added_2026-10-26/`): **Trafalgar Group, Oct 21–23 2026** ("Maine General Statewide
Survey", 9-page PDF dated October 2026, posted Oct 25; 1,077 likely general-election voters, MoE 2.9, response rate 5.4%) — U.S.
Senate Collins 47.0 / Jackson 46.5 / undecided 6.5 (printed "Troy Dale Jackson", stored as option "Troy Jackson"; listings round
to 47 / 47). Senate only; grade A (the report carries age, gender and ethnicity breakdowns of the vote — not added, only party
tables are structured; its party table is sample composition only: 37.4 D / 31.7 R / 30.9 no party/other). Pollster class and
type copied from the earlier 2026 Trafalgar poll (P2026-1001, grade C from its news post). 685 polls (+1), 2,144 questions (+1),
7,318 results (+3). A 2026 question: `poll_question_links.csv` got the same "no contest in results (not yet held)" row, so no
link or accuracy row changed. Rebuild: as for v0.4.3, then unzip `build/polling_patch_2026-10-26.zip` (or, from v0.4.3 `data/`,
just `python3 scripts/apply_additions.py`). Not updated: `pollsters.csv` counts.

**v0.4.3 (2026-10-23, scheduled freshness check C — polls)** adds two Maine polls through `additions/`
(`make_additions_2026-10-23.py` names the source of every figure; patch `build/polling_patch_2026-10-23.zip`), each read from
the pollster's own document (copies in `sources/added_2026-10-23/`): **Fox News Poll (Beacon Research / Shaw & Co.), Oct 16–20
2026** (topline PDF released Oct 22; 1,514 likely voters ±2.5 of 1,689 registered ±2; grade B) — U.S. Senate Jackson 49 / Collins
49 / other 1 / don't know 1; Governor Pingree 49 / Charles 39 / Bennett 9 / don't know 2 (likely-voter figures, Fox's headline;
the registered-voter figures are in each question's `context`; "*" cells under 0.5% not recorded); pollster class and type copied
from the two earlier 2026 Fox polls so the three are treated alike. **Tulchin Research (D), CD 2, Oct 12–14 2026** (memo dated
Oct 16; 500 LV ±4.38; grade B) — Dunlap 50 / LePage 46 / undecided 4 (the memo was unreadable on Oct 19–22; field dates now from
the memo). 684 polls (+2), 2,143 questions (+3), 7,315 results (+11). Both are 2026 questions: `poll_question_links.csv` got the
same "no contest in results (not yet held)" rows that `link_results.py` writes for them, so no link or accuracy row changed
(`link_results.py` was not re-run; P-243 asks for a full re-run after results v3.18). Rebuild: as for v0.4.2, then unzip
`build/polling_patch_2026-10-23.zip` (or, from v0.4.2 `data/`, just `python3 scripts/apply_additions.py`). Not updated:
`pollsters.csv` counts. Fox crosstabs (party tables) not added.

**v0.4.2 (2026-10-17 afternoon, Matt: "Robert Charles is obviously Bobby Charles")** clears the `hypothetical` flag on nine
2026 general-election questions whose named candidates are all on the SOS November 2026 ballot for that office: Beacon/Shaw
Jun 27 (Governor) and Aug 6–10 (Senate, Governor), UMass Lowell/YouGov Sep 14 (Senate, Governor) and YouGov Sep 2–8 (four
questions). The flag came from the listings (Wikipedia's "hypothetical" tables, kept after Jackson replaced Platner, and
names printed as 'Robert "Bobby" Charles' / "Richard Bennett"). The rule — same surname, and the same first name, a common
short form of it or the nickname printed in quotes, checked against `campaign-finance/raw/sos-nov2026/nov2026_candidates.csv`
— is in `corrections/make_nominees_2026-10-17.py`; its output `corrections/nominees_2026-10-17.csv` (`set_question … hypothetical
no`) is applied by `apply_corrections.py` on every rebuild. Questions naming anyone not on the ballot (Platner, Mills,
Baldacci …) keep the flag. No figure changed; options keep the names as printed. FLAGS §18.

**v0.4.1 (2026-10-17, scheduled freshness check C — polls)** adds the two Maine polls released since v0.4, each from the
pollster's own publication, through `additions/` (`make_additions_2026-10-17.py` names the source of every figure; patch
`build/polling_patch_2026-10-17.zip`): **Suffolk University, Oct 8–11 2026** (500 LV; marginals PDF, grade B) — U.S. Senate
Jackson 49.0 / Collins 47.0 / undecided 3.8 / refused 0.2; Governor Pingree 49.4 / Charles 38.0 / Bennett 7.6 / undecided 4.6 /
refused 0.4 (vote-or-lean wording; printed "Robert Charles", stored as option "Bobby Charles" like the other 2026 general
polls); **InsiderAdvantage, Oct 13–14 2026** (1,200 LV, MOE 2.83, mixed mode text/panel; release of Oct 15, grade C) —
U.S. Senate Jackson 49 / Collins 47 (the release prints no undecided/other and its tabs were not yet posted, so only those
two figures are on file). 682 polls (+2), 2,140 questions (+3), 7,304 results (+11). Both are 2026 questions, so
`link_results.py` gives them "no contest in results (not yet held)" and every other link/accuracy row is unchanged.
Rebuild: as for v0.4, then unzip `build/polling_patch_2026-10-17.zip` over the build (or, from v0.4 `data/`, just
`python3 scripts/apply_additions.py`), then `link_results.py` with results v3.14. Not updated: `pollsters.csv` counts
(`n_maine_polls`, `last_year`) for Suffolk and InsiderAdvantage — `apply_additions.py` does not recount them.

**v0.4 (2026-10-16, party crosstabs from model-data; PENDING-WORK P-190)** adds what the model-data crosstab pass of
2026-10-16 found in reports v0.3.2 had not structured: **2,653 party crosstab cells for 169 questions in 54 polls**
(UNH/Press Herald 2014–2016 by self-reported registration and party ID, NYT/Siena 2018 and 2026 by voter-file registration,
14 SurveyUSA reports 2006–2024, Suffolk, Quinnipiac, Colby, Emerson, PPP, Ipsos, Fox, CNN, Washington Post and others), and
**24 questions with their toplines** that v0.3.2 lacked: the **Magellan Strategies poll of Oct 23–24 2014** (a new poll:
LePage 42, Michaud 42, Cutler 13), PPP's three-way Oct 2014 governor table, UNH 2014 governor/Senate questions with and without
leaners, UNH Sept 2016 four-way President, SurveyUSA's 2006 Taxpayer Bill of Rights (three waves), 2021 CMP-corridor and 2024
state-flag questions, and generic ballots (2014 Maine House and Senate, 2024 Legislature, 2022 and 2026 U.S. House; office
`STLEG` = generic Maine Legislature ballot, flagged `generic opponent/ballot`, never charted or scored).
**680 polls, 2,137 questions, 7,293 option results, 21,937 crosstab cells.** The Magellan release (doc 11fee5d790) had been
filed under the Ipsos/BDN poll P2014-1029-ipsos and "Magellan Strategies" listed as an Ipsos alias; both moved to the new poll
(pollsters.csv gains Magellan Strategies, Republican-aligned firm). Accuracy table: +1 row (Magellan, 2014 Governor).
How it is built: `additions/` holds hand-checked tables (`polls_`, `poll_questions_`, `poll_results_`, `poll_crosstabs_2026-10-16.csv`,
written by `additions/make_additions_2026-10-16.py`, which names the source of every topline); `scripts/apply_additions.py`, now
the last step of `run_all.sh` after `apply_corrections.py`, adds them to `data/` without ever overwriting a record the build
already has, and matches each crosstab option label to the question's own labels ("Trump" → "Donald Trump"; the printed label is
kept in `notes`). Rebuild: as for v0.3.2, then unzip `build/polling_patch_2026-10-16.zip` over the build and run `run_all.sh`
(or, from v0.3.2 `data/`, just `python3 scripts/apply_additions.py`), then `MPD_ER=<unzipped results v3.14> python3 scripts/link_results.py`.
Applying the patch to the committed v0.3.2 data reproduces v0.4 exactly; no v0.3.2 record changed except P2014-1029-ipsos's document list and the Ipsos row's aliases (links and accuracy tables only gain rows).


This dataset covers public opinion polls on Maine elections for **President, Governor, U.S. Senate, U.S. House (CD1 and CD2)** and **statewide ballot questions**. It includes general elections and primaries, and hypothetical match-ups that belong to a specific election. Campaign, party and partisan polls are included and labelled by poll type. The plan is in `PLAN-v1.md`.

**v0.3.2 (2026-10-15, newspaper pass 2 finished)** adds the last 441 newspaper articles of pass 2 (batches b0409–b0503,
every search page now read) on top of v0.3.1: **679 polls (313 before 2004; was 669 / 303), 2,113 questions, 7,198 option
results.** New polls include Capitol News Service's 1990 rail-bond poll, a 1987 Guy Gannett Maine Yankee poll, the 1986 Tierney
(Kiley) and Huber campaign governor polls, 1978 Frankland campaign figures, the 1983 Sportsmen's Alliance moose-hunt poll, the
1989 nuclear-waste compact (Question 13) and 1998 R&D-bond polls, the Sept. 1995 Bullet Poll wave on Question 1 and the 1998
Democratic governor primary Bullet Poll; 41 more questions join polls on file (CNS 1988–1990, Mason-Dixon 1996,
Critical Insights 2000, SMS 2001 bonds, RKM 2000, 1974 and 1980 university polls, 1981–82 Senate match-ups). Two old entry errors
are fixed in the transcriptions (a 1990 and a 1995 poll whose two waves had been recorded as one). Details and the five pre-2004
poll ids that changed: FLAGS §9 and §15. Rebuild: as for v0.3.1, then unzip `build/polling_patch_2026-10-15b.zip` over the build
(new and corrected `news/out/` batches, `news/cleanup_pass2_b0409-b0503.csv`, `corrections/pass2_final_2026-10-15.csv`).

**v0.3.1 (2026-10-15, election results v3.14)** re-links the polls to election results v3.14 and scores ranked-choice
races against the official totals (Matt: "Make the totals the official totals"). `scripts/link_results.py` (in
`build/polling_patch_2026-10-15.zip`; it replaces the copy in `polling_build_v0.3.zip`) takes, for the 11 contests whose
`contests.csv` `headline_total` is `official_rcv_round1`, each candidate's figure from the Secretary of State's official
round-one tabulation instead of the sum of the town rows. Only `poll_accuracy.csv` and `pollster_maine_accuracy.csv` change:
2024 U.S. House District 2 actual margin 0.58 → 0.69 points (Golden 197,151, Theriault 194,445), the 2026 Governor and
District 2 primaries by a few hundredths, 1974 Governor (Longley 142,464, the official figure after results v3.13's
rule-19 correction) 3.72 → 3.73 two-way, and 1976 U.S. Senate 20.38 → 20.41. Links, questions and results are unchanged.
Rebuild: unzip `polling_build_v0.3.zip`, then `polling_patch_2026-10-14.zip` and `polling_patch_2026-10-15.zip` over it,
copy `data/` and `corrections/` in, and run `MPD_ER=<unzipped results> python3 scripts/link_results.py`.

**v0.3 (2026-10-14, Data Audit II fixes)** cleans the data rather than adding polls:
- **Newspaper reprints and recalls merged.** The same survey printed again days or weeks later (or recalled as "last month's poll")
  had been loaded as a separate, later poll. 105 such questions and 106 questions printed twice inside one poll are now merged into the
  earliest record with its true dates (audit PC-4). 154 single-figure fragments are dropped. Every removed record is listed, with the
  record it duplicates and the reason, in the new table `data/poll_reprints.csv`.
- **Listing copies beside the document copy removed** (PC-1, PC-3): Pan Atlantic May 2026 (David Jones 20% listing copy), Emerson Sept 2016 CD2,
  Siena Sept 2018 CD2, Pan Atlantic Oct 2022 CD1, and the GQR Oct 2014 row that carried PPP's May 2014 figures.
- **Pan Atlantic March 2026 CD1/CD2 figures removed** (PC-2): the March report repeats the December sentence word for word; the figures do not
  fit the March statewide result.
- **Same poll entered twice merged:** Critical Insights fall 2023 and spring 2022, Colby July 2020 (three records), Mason-Dixon/BDN Oct 1996,
  BDN 2026-06-03 = SurveyUSA, CNS/Northeast Research May 1986, Sun-Journal/Critical Insights Oct 2000, and others (see `poll_reprints.csv`).
- **Flags** (PC-5, PC-7, PC-8): subgroup figures (men, women, one party, one region, undecided-only) are flagged `subgroup only` and the 1972 BDN
  11-county surveys `partial-state sample`; 19 questions whose printed figures add to 106% or more are flagged `figures do not add up`;
  post-election recalls with unknown dates are flagged. None of these are scored for accuracy.
- **Ranked-choice "round" rows** that had been loaded as answers are removed; later-round questions carry the `ranked-choice item` flag (PC-6).
- **Dates and links** (PC-8): post-election recaps re-dated to the poll they repeat; UNH July 2026 convention question unlinked from the June primary;
  UNH Oct 2025 CD2 question moved to election year 2026; FAU 2024 marked as the Republican primary.
- **Pollster names** (PC-9): Marttila & Kiley spelled correctly; Colby/SocialSphere, Telegram, Guy Gannett and Goldsmith variants merged; the two 1994
  Press Herald polls credited to UNH are now "Unnamed pollster for Portland Press Herald".
- **Ballot questions linked to results:** 348 of 384 referendum poll questions now link to their referendum contest and question number
  (`poll_question_links.csv` columns `ref_question`, `ref_description`). 27 more are 1990–1994 or 2026 questions not yet in the results.
- **Labels:** candidates named Donald (Trump) were left out of question labels; fixed in `build_final.py`.
- Accuracy: `poll_accuracy.csv` now has 264 rows (was 298); recalls, subgroups and double-counted polls are gone. `pollster_maine_accuracy.csv` recomputed.

**How the fixes are kept:** `run_all.sh` now ends with `python3 scripts/apply_corrections.py`. It applies the hand-checked list in
`corrections/audit_2026-10-14.csv` and the reprint/duplicate rules (described in `corrections/README.md`) every time `data/` is rebuilt,
including when new newspaper batches are added. Run `scripts/link_results.py` after it, as before. Do not edit `data/*.csv` by hand:
put a row in a corrections file instead.

**v0.2 (2026-10-09 evening)** added: the newspaper sweep (first results — see "Newspapers" below), crosstabs from the PPP-style reports, links from every question to the election-results contest and candidates, a Maine accuracy table per pollster, the latest 2026 polls, and fixes (population column, ranked-column primaries, margin rows, sponsor/poll-type rules). v0.1 was the first collection pass. It is complete for what the online sources hold from 2004 to 2026. It is thin before 2004, where the records are mostly in newspapers; see "Gaps" below and FLAGS.

## Tables (folder `data/`)

| File | One row per | Key columns |
|---|---|---|
| `polls.csv` | poll (one fieldwork period by one pollster) | `poll_id`, `pollster`, `pollster_class`, `sponsor`, `poll_type`, `partisan_flag`, `sponsor_candidate`, `field_start`, `field_end`, `max_sample_n`, `races`, `document_grade`, `document_files`, `listing_sources`, `reprints` (records merged into this poll) |
| `poll_questions.csv` | question (one race or ballot question; CD1 and CD2 subsamples are separate questions) | `question_id`, `poll_id`, `election_year`, `office` (PRES/GOV/SEN/HOUSE/REF), `stage`, `party`, `district`, `question_label`, `population`, `sample_n`, `moe`, `hypothetical`, `flags`, `verified_from_document`, `sources` |
| `poll_results.csv` | answer option | `question_id`, `option_as_printed`, `option`, `candidate_party`, `pct` |
| `poll_crosstabs.csv` | crosstab cell (v0.4: +2,653 party cells from `additions/`) | `question_id`, `source_doc`, `page`, `group_dim`, `group_value`, `group_n`, `option`, `pct`, `notes` |
| `pollsters.csv` | pollster | `pollster_class`, `maine_based`, `n_maine_polls`, years active, latest FiveThirtyEight and Silver Bulletin grades, aliases, notes |
| `poll_question_links.csv` | question | `question_id`, `contest_id` (election results v3.11 or later), `link_status`; for ballot questions `ref_question` (number) and `ref_description` |
| `poll_reprints.csv` | removed record (v0.3) | `kind` (question/poll), `id`, `reprint_of`, pollster, dates, sources, figures, `reason`, `audit_id` |
| `poll_option_links.csv` | answer option | `question_id`, `option`, `person_id`, `candidate_display`, `contest_id` |
| `poll_accuracy.csv` | poll × contest (final 21 days) | first/second finisher, poll and actual raw margins, **two-way margins** (top two only, so undecideds are not counted as error), `two_way_margin_error`, `lean_dem` (+ = poll more Democratic than the result), `called_winner` |
| `pollster_maine_accuracy.csv` | pollster | `n_polls`, `mean_abs_two_way_margin_error`, `mean_lean_dem`, `pct_called_winner`, years |
| `pollster_ratings.csv` | pollster × rating source × edition | FiveThirtyEight editions 2014, 2016, 2018, 2019, 2020, 2021, 2023, 2024-25; Silver Bulletin June 2024, Feb 2025, Jan 2026. Grade, predictive plus-minus, mean-reverted bias (positive = Democratic), AAPOR/Roper flag, banned flag |

Coverage:
- v0.3.2 (2026-10-15): 679 polls (313 before 2004), 2,113 questions and 7,198 option results; newspaper sweep passes 1 and 2 finished (b0000–b0503).
- v0.3: 669 polls (304 before 2004), 2,069 questions and 7,050 option results after the reprint merge (v0.2 had 773 polls, 2,454 questions, 7,980 results, built from the same sources on 2026-10-14 (newspaper sweep pass 1 finished 2026-10-11, b0000–b0254; pass 2 — the cut-short and missing searches, see FLAGS §9 — under way since 2026-10-13, batches to b0408 transcribed).
- 613 of the questions (v0.3) were read directly from the pollster's own document. The rest come from listings that agree with each other.
- 19,284 crosstab cells for 258 questions (v0.3): University of New Hampshire reports (69 questions, parsed), PPP-style reports (108 questions, parsed from the PDFs' "Crosstabs" pages and matched to the question by its toplines) and 82 more questions transcribed by hand from Pan Atlantic, MPRC, Colby, Critical Insights, SurveyUSA, Siena, Marist, Quinnipiac, Suffolk, Data for Progress, Fabrizio and YouGov reports (`crosstabs/manual_xt/`, method `crosstabs/XT_INSTRUCTIONS.md`; a `notes` column marks partial groups).

### Poll types
Each poll is assigned one of these types:
- Public (pollster-released)
- University / academic
- Media-sponsored
- Partisan (pollster or sponsor aligned with a party)
- Campaign internal
- Party committee
- Advocacy / interest-group sponsored
- Interest-group sponsored (bipartisan team)

These come from the listings' flags ("(D)", "(R)", "(I-Hayes)", 538's internal and partisan fields), the named sponsor, and the pollster's class. Pollster classes are Claude's judgment from each firm's known clients, kept in `scripts/pollster_classes.py`; Matt can change them.

### Document grades
- **A**: full report with crosstabs or breakdowns.
- **B**: toplines report.
- **C**: news story or press release.
- **D**: no document found yet; the figures come only from aggregator listings.

The source documents (447 files) are in `sources/source_documents_part1.zip` to `part8.zip`. Each file is named by the `key` in `sources/doc_links.csv`, which gives its original web address, its poll(s) and how the link was made. The full build kit (scripts, listings, extracted text, hand transcriptions, corrections) is `build/polling_build_v0.3.zip` (v0.3; `polling_patch_2026-10-14.zip` holds only the files changed since v0.2); `work/polling_work_checkpoint.zip` is an earlier checkpoint and can be deleted.

### Question flags
`hypothetical` = yes, no or unknown:
- **yes** if a named candidate was not among the candidates polled in the last 45 days before that election.
- **unknown** for past races where no late polls exist.

`flags` can contain:
- election-day exit poll (kept, left out of the accuracy table)
- subgroup only (not a statewide topline) (v0.3)
- figures do not add up (check source) (v0.3)
- figures may mix two survey waves (check source) (v0.3)
- recalled after the election (field dates unknown) (v0.3)
- earlier poll recalled in a later article; only the month is printed (v0.3)
- partial-state sample (a newspaper's own circulation area or one region)
- generic opponent/ballot
- asked after messaging (not a clean topline)
- candidate support-likelihood battery (not a ballot)
- replacement-nominee question
- issue question used as ballot-question proxy
- includes leaners
- ranked-choice item

## How it was built (scripts in `scripts/`)
1. **Listings.** Each was saved and normalized by `norm_sources.py`:
   - Wikipedia poll tables: 157 pages, plus the Maine sections of the national statewide-polling pages (`wiki/`).
   - RealClearPolling: 72 Maine race pages (`rcp/`).
   - The FiveThirtyEight poll database 2018–2025, recovered from the Internet Archive after 538 closed (`f538/`).
   - The FiveThirtyEight and Silver Bulletin pollster-rating poll files, 1998–2024.
   - electoral-vote.com files, kept as a cross-check only (`raw/`).
2. **Merging.** `build_register.py` merges the same poll across listings:
   - Same pollster, end date within 3 days, and identical figures.
   - Pollster names are normalized in `pollsters.py`.
3. **Documents.** Pollster reports were fetched from:
   - the pollsters' sites;
   - the Internet Archive, by searching archived PDF listings for panatlanticsmsgroup.com, criticalinsights.com, digitalresearch.com, mprc.me, colby.edu and publicpolicypolling.com;
   - the UNH scholars repository (all 57 Maine reports).

   `map_docs.py` links each document to its poll.
4. **Transcription from documents.**
   - UNH reports were parsed automatically, including crosstabs (`parse_unh.py`, `unh_questions.py`).
   - About 160 other reports (PPP, Pan Atlantic, Critical Insights, MPRC, Colby, SurveyUSA, Emerson, Suffolk, Siena, Fox, Marist and others) were transcribed by hand from the text and rendered pages. Each figure carries the exact source text it came from (`manual/out/<doc>.csv`, rules in `manual/INSTRUCTIONS.md`).
   - Document figures take precedence over listings.
5. **Final tables.** `build_final.py` and `build_crosstabs.py` write `data/`.
6. **Corrections (v0.3).** `apply_corrections.py` applies `corrections/*.csv` and the reprint/duplicate rules (last step of `run_all.sh`).
7. **Links and accuracy.** `link_results.py` (env `MPD_ER` = unzipped election results) writes the link and accuracy tables.

In the cloud copy of the build kit, `norm_sources.py` and `map_docs.py` cannot run (they need the raw 538 files and `raw/unh_index.json`, which are not in the kit); their outputs (`listings/questions_long.csv`, `register/doc_links.csv`) are in the kit, so the build runs from `build_register.py` onward.

## Gaps (see FLAGS for detail)
- **Before 2004:** 401 polls as of 2026-10-14 (newspaper sweep pass 2 under way) (newspaper sweep; was 23 aggregator lines before it). 1990s and earlier polls survive mainly in newspapers. This needs Newspapers.com in the Claude browser on the iMac (P-145).
- **Pan Atlantic Omnibus:** the series has run twice a year since 1996 (about 69 editions). Political results are on file for 2007, 2008, 2009, 2014, 2020, 2022, 2024, 2025 and 2026. Asked for the rest by email (P-145).
- **Critical Insights on Maine:** the series runs spring and fall since 1998. Reports are on file for 2005, 2010, 2013, 2014, 2022, 2023 and 2024. The 2006–2012 archive links point to request forms rather than the PDFs. Asked by email.
- **Crosstabs:** only UNH crosstabs are parsed so far. The other A-grade reports hold crosstabs in PDFs and are in `sources/` but not yet in tables.

## Newspapers (Newspapers.com, v0.2 onward)
Matt's Newspapers.com subscription (Publisher Extra) is searched through his Chrome on the iMac. The collector
(`news/harvest.js`, run inside a signed-in newspapers.com tab) searches the Maine papers (Bangor Daily News, Portland Press Herald,
Maine Sunday Telegram, Evening Express, Kennebec Journal, Morning Sentinel, Lewiston Sun-Journal / Daily Sun, Biddeford Journal, Times Record…)
for 1960–2012 with (a) "poll undecided" / "survey undecided" for every year, (b) "poll" + "per cent"/"percent" + the two leading
candidates' surnames for every statewide and congressional contest 1960–2003, (c) "poll referendum per cent" per year and (d) pollster names
(Strategic Marketing Services, Pan Atlantic, Critical Insights, Command Research, Market Decisions, Northeast Research, Becker, "Maine Poll", "Press Herald poll", "Bangor Daily News poll"…).
For each page it reads the newspaper's own article segmentation and OCR text, keeps only the segments where the search terms fall,
and keeps those mentioning a poll with figures. A filter keeps articles naming two candidates of that year (or a ballot question) with figures, and drops
national (Gallup/Harris) stories and near-duplicate AP copies. Claude agents then transcribe each batch following `news/INSTRUCTIONS.md`
into `news/out/<batch>.csv` (every figure with the OCR fragment it came from) and `<batch>_log.csv` (a verdict for every article).
`scripts/news_to_listing.py` turns them into `listings/news_doc_questions.csv` (source `news`; priority after Wikipedia/RCP and the pollster's own documents).
Newspaper-sourced polls have document grade **C** and link to the page image (`https://www.newspapers.com/image/<page id>/`).
Where a story prints no field dates, the end date is the day before publication (noted in `date_raw`).
Newspapers.com limits request rates, so the collector runs at about one request a second; the full sweep takes a day or more.
**Status:** the sweep finished on 2026-10-11 (started 2026-10-09): 18,664 search-result pages, 6,353 article segments saved, 255 batches transcribed. 101 page requests failed during the run and were not retried (counted by the collector; it keeps no list of which pages).

How newspaper questions become polls (`scripts/news_to_listing.py`, `build_register.py`):
- one question per article; the same result reprinted in another article or paper (identical figures for the same race) is kept once, the
  named-pollster/dated copy preferred, with every printing's page link kept;
- no field dates printed: end date = day before publication, or earlier when the story says so ("three weeks ago", "last month",
  "an earlier poll" ≈ 3 weeks); a poll of an earlier election recalled in a later story is not loaded (no usable date);
- questions are grouped into polls by pollster and end date within 2 days of the poll's first question (4 days when the dates come from
  publication) — anchored, not chained, so daily stories no longer merge separate waves (this split one 2020 SurveyMonkey tracking window
  into two polls).

## Question ids
`question_id` values are kept stable across rebuilds by `register/question_id_registry.csv` (signature = poll, race, option labels and figures → id).
v0.2 also drops listing-only questions that duplicate a document question of the same poll (69) and options printed only as "%" (column-header debris).