Methodology
How Civio summaries are made
This page is intentionally specific. We’d rather you find a problem with our methodology than discover one with a summary. Everything on this page is verifiable. When the pipeline changes, this page changes — commit-by-commit.
The pipeline
Ingest only from .gov
Bill text comes from Congress.gov (federal), OpenStates v3 (50 states), and the official meeting agendas / minutes of city councils, school boards, and county boards via the LCIM crawler. We never ingest news media, opinion pieces, or secondary sources as input to a summary. If the bill is not in an official government feed, it does not enter the pipeline.
Generate a draft with Claude
A constrained Claude prompt produces (a) a one-sentence headline, (b) three plain-English bullets, and (c) a six-section impact analysis (who it affects, money flow, agencies, timeline, related laws, what it changes). The prompt is checked into the repo at lib/civio-analysis-prompt.ts and reviewed in every PR that touches it.
Strip framing language
A second pass scans the draft for opinion words ("dangerous," "common-sense," "extreme," "radical," "necessary," "burdensome"), partisan idioms, and emotional adjectives. Any match is regenerated. Summaries containing first-person voice or rhetorical framing are rejected and re-run.
Verify provenance
Every claim in a summary must trace to a sentence in the bill text. We embed sentence-level citations in the underlying record. If a claim cannot be traced, the summary is downgraded to "headline only" until a human reviewer expands it.
Charged-category review queue
Summaries categorized as touching abortion, immigration, firearms, election law, or healthcare coverage are held in `summary_review_status = pending_review` and do NOT publish to users until a human reviewer approves them. This is enforced at the database level (see migration 005) — it is not a settings flag that can be quietly disabled.
Publish with a flag affordance
Every published summary carries a visible "Flag this summary" button. Flags create a record in `summary_flags` and surface in our internal review queue within 60 seconds. Our target review SLA is 24 hours. Resolved flags are linked to any resulting correction.
What we human-review before publication
These categories carry the highest cost of mischaracterization. Summaries in these categories are held in a review queue and require a named human reviewer’s approval before they are shown to users.
This list will grow. Additions are recorded in the public commit log.
Words our pipeline rejects
Plain-English does not mean editorialized. The framing-pass blocks the following words in published summaries. If a draft contains one, it is regenerated. (Both partisan flavors of framing language appear here on purpose — this is not a one-sided filter.)
How we measure ourselves
Accuracy eval set
A held-out set of bills with human-written reference summaries. We run the eval before every model or prompt change and publish the agreement score with the reference. If the score drops below the previous baseline, the change is blocked from merging.
Current eval size: building (target: 200 bills before July launch)
Partisan-balance audit
We coded every bill by sponsor party and run sentiment / word-choice measurements on the summaries by party. If the delta between Democrat-sponsored and Republican-sponsored bills exceeds a fixed threshold for any tracked metric, the prompt is reviewed and we publish the issue.
Last audit: building (will be published before July launch)
Flag rate
The most important real-world signal. We publish flags submitted, flags resolved-with-correction, flags resolved-no-change, and median time-to-review. These numbers will be updated weekly on this page after launch.
What Civio does not do
- —We do not characterize legislation as “good” or “bad.”
- —We do not predict political outcomes (will it pass? who will benefit politically?).
- —We do not amplify or downrank summaries based on user political signals. There is no political profile.
- —We do not feed user demographics back into the summary generator.
- —We do not let advertisers, donors, or B2B customers see drafts before they are published.
How to challenge a summary
- Tap “Flag this summary” on the page where it appears.
- Pick a reason: Inaccurate, Missing context, Politically framed, or Other.
- Add a short description of what’s wrong and, if you can, the sentence in the bill text you’re comparing it to.
- A human reviewer will look at it. If a correction ships, you’ll see a notice on the bill and the entry will appear in the public corrections log.
For something more structured, email corrections@civio.app.