The 990 Lab: how to verify
Freeze an AI summary of a real nonprofit's Form 990, then catch every error against the filing. The method behind Rail B's funding lab, and the filing rules behind Rail A's comparison.
This is the course’s signature discipline: you pull a real organization’s public Form 990, ask a chatbot to summarize its finances, then go line by line and catch everything the AI got wrong, citing the filing that contradicts it.
Not a separate grade. The freeze-and-verify is graded in Rail B’s B2 funding lab (one comparable org per teammate) and powers every AI process log; the filing rules and citation standard here also govern Rail A’s A3 comparison. We build it all in lab week (Week 8), and everything after that week assumes you have it.
Pick a filing that can be checked
- Any real nonprofit’s filing works, found on ProPublica Nonprofit Explorer. A slightly larger org usually gives you more to check.
- It has to be a full Form 990 or a 990-EZ. The 990-N e-Postcard that the smallest nonprofits file carries only a handful of fields, not enough to check a summary against.
- Record the org’s legal name and EIN, a direct link to the filing, and the tax year: the months on page 1, item A. 990s run a year or more behind; mixing years can make an accurate figure look wrong.
Freeze the AI summary
Ask a chatbot to summarize the organization’s finances. Then freeze it: paste the exact prompt and the exact output, unedited.
Then number every factual claim it made. A claim is anything checkable: a revenue figure, an expense ratio, a named grant or funder, a headcount, a founding year, a program description stated as fact. Most summaries yield somewhere between eight and twenty.
You need at least eight. If yours comes back thinner, ask again for a fuller summary covering revenue, expenses, staffing, and programs, then freeze that one and number it. A summary too vague to produce eight checkable claims is worth noticing in its own right: it’s a summary nobody could hold to account.
Verify, one row per claim
| # | The AI’s claim | Verdict | The 990 says | Where |
|---|---|---|---|---|
| 1 | quote it | Correct / Wrong / Unverifiable | the actual figure or fact | Part & line, and the PDF page |
Verdict rules:
- Correct means you found it in the filing and it matches. Cite it anyway.
- Wrong means the filing says something different. Give the real number and where you found it.
- Unverifiable means the 990 simply doesn’t carry that fact. This is a real and common finding, and it is not a free pass: say what you would need to check it, and check the org’s own site or audited statements if they have them.
Every row needs a location. “Part IX” is not a location; “Part IX, line 25, column A, p. 10” is. A row without a citation counts as unchecked.
Eight rows is the floor in the funding lab, and twelve plain rows beat three beautiful ones.
The errors worth hunting
A total-revenue figure that isn’t the filing’s total revenue. An invented program-expense ratio. A grant that appears nowhere in the filing. A confident statement about staff or compensation that the 990 doesn’t support. A number silently taken from a different tax year.
Why this exists
The rest of the course asks you to write about real organizations using real numbers. This is where you find out, concretely, how confidently a language model will hand you a wrong one, and what it takes to catch it. That reflex, not the table itself, is the deliverable.