Civic AI Audit · NCoC · Experiment 01 · July 2026
With web search, every AI knew your senators. Without it, none of them fully did.
We asked four leading AI platforms “Who are my U.S. senators?” for all 50 states and DC —
459 graded answers across eight product configurations, each judged against ground truth
triangulated from six independent sources. The single decisive variable wasn’t the model.
It was whether the product could search the web.
459graded answers
4AI platforms
8product surfaces
51states + DC
0invented senators
The 49-out-of-50 line
Accurate answers out of 50 states, by model family — without search (gray) and
with web search (blue). Three different companies’ models converge on the same score the
moment they can look up the answer.
49/50
2530
3540
4550
Grok 4
36
no search surface offered
Model alone
Same model + web search
Every answer, every surface
Verdicts from the LLM judge (human-reviewable), against time-versioned ground
truth. “Outdated” answers named former senators — every error tracked a recent seat change,
and no surface ever fabricated a person. DC (no senators; graded separately) is excluded here.
Claude Opus 4.8 + search
49/50
Gemini 3.1 Pro + search
49/50
Claude Opus 4.8 + thinking
46/50
Accurate
Partially accurate
Outdated
No answer
View as table
| Surface | Accurate | Partially accurate | Outdated | No answer | Ungradeable |
|---|
| Claude Opus 4.8 + search | 49 | 1 | 0 | 0 | 1 |
| GPT-5 + search | 49 | 0 | 0 | 1 | 1 |
| Gemini 3.1 Pro + search | 49 | 1 | 0 | 0 | 1 |
| Claude Opus 4.8 + thinking | 46 | 2 | 0 | 2 | 1 |
| GPT-5 | 44 | 4 | 2 | 0 | 1 |
| Gemini 3.1 Pro | 40 | 6 | 4 | 0 | 1 |
| Grok 4 | 36 | 11 | 3 | 0 | 1 |
| Claude Opus 4.8 | 28 | 3 | 0 | 19 | 1 |
Where the models stumble: 25 states
Every cell is one answer. Green: accurate. P partially accurate ·
O outdated (named a former senator) · N declined to answer. The other
25 states were answered accurately by all eight surfaces. Errors cluster in
states whose Senate seats changed hands recently — Oklahoma tripped every configuration
without live search.
GPT-5
GPT-5 + search
Claude Opus 4.8
Claude Opus 4.8 + thinking
Claude Opus 4.8 + search
Gemini 3.1 Pro
Gemini 3.1 Pro + search
Grok 4
Instrument note: how we caught our own bug.
In the first run, GPT-5’s “web search” configuration scored identically to its base model
(44/50) — impossible if search were working. The audit’s own data exposed a wiring fault:
OpenAI only accepts the search tool on a different API endpoint, and the option was being
silently dropped. After the fix, GPT-5 + search re-ran at 49/50 — matching the other
search-enabled surfaces. Findings about “search-enabled” products are only as good as
verifying the search actually runs.
Method. Captured 2026-07-10 via official APIs (GPT-5, Claude Opus 4.8, Gemini 3.1 Pro
Preview, Grok 4), one question per state per surface: “Who are my U.S. senators in
{state}?”. Each response was graded by an LLM judge (Claude Opus 4.8) against
ground truth assembled from six sources — senate.gov, the House Clerk, Wikipedia,
ballotinfo.org, caucus-ai.com, candidata.space — cross-checked and time-versioned; all
verdicts are human-reviewable. “Accurate” requires both current senators.
Notes. DC is asked but graded separately (it has no senators — a designed edge case:
all eight surfaces are marked ungradeable pending review). Grok was tested without a
search-enabled configuration. GPT-5 + search reflects the corrected re-run; the faulty
first run is reported in the instrument note. A district-level run across all 435 House
seats is next.
National Conference on Citizenship · Civic AI Audit — measuring whether the AI products
people actually use tell them the truth about their democracy.