How we check papers — and what we can't

CoChat runs several checks on a paper. Some are computed. Some use a model. This page says what each one does, what it cannot see, and how often it is right where we have measured it. The report on a paper and your Library drawer link here with How we check this.

Nothing here is a peer review. A check can be wrong in both directions: it can flag a correct paper, and it can miss a real problem.

Is the paper real?

We look the DOI up in the publisher registries (CrossRef and DataCite) and compare the title and year. The badges are explained in Papers and verification.

What it checks: that the DOI exists and points at this paper. It does not read the paper.
Unverified means the lookup failed (a timeout or a rate limit). It is looked up again in an hourly pass. A failed lookup never turns into Not found.
Preprints without a published version are looked up again about every two weeks.
Retractions: once a day we read CrossRef’s feed of retraction, withdrawal and expression-of-concern notices, and mark every paper a notice names. A paper is shown as retracted when CrossRef or OpenAlex says so. A retraction that neither records is invisible to us.

Claims the paper makes

A model reads the paper and proposes its claims. A claim is kept only when its quote appears word for word in the text. A second model then labels each claim against our definition. Those labels decide which claims a report shows first and which it shows collapsed, with the reason. Collapsed claims are never hidden from a signed-in reader.

The labels are not yet calibrated, so no card shows a confidence band. See How we measure confidence.

Do the cited sources say it?

When a claim cites a source, we check whether that source says what the claim relies on. The same check runs when you Verify a claim you added.

A verdict comes only from the source’s full text that CoChat holds. It never comes from an abstract or a search snippet.

A direct quote is looked up word for word in our copy of the source. No model is involved.
Otherwise a model judges it on our copy of the source, and on nothing else.

No full text, no verdict. When we don’t hold the source’s full text, nothing is checked and nothing counts against your daily claim checks. We try to fetch the text. Meanwhile the claim reads Fetching the cited paper…, and when no full text can be found it reads Not checked — no full text of the source.

A paper’s own references are checked automatically once the text arrives.
A claim you verify yourself (Verify in the Library, or in a literature review) keeps Fetching the cited paper… until you click Verify again once the text is in.

A check whose model gives no verdict doesn’t count against your daily claim checks either. For now most cited sources have no full text with us, so most references stay unchecked.

Verdicts are confirmed, partly supported, disputed, misattributed and unverifiable. Partly supported means the source backs part of the claim, or backs it for a different population, condition or comparison; the verdict names the part it doesn’t back. A claim split into parts is also partly supported when only some parts are backed. A citation that only names something the paper uses (a questionnaire, a dataset, a study it pooled) is checked on whether the source is that thing. When the model is not confident (below 0.40 on its own scale), the verdict is unverifiable, never disputed.

What it cannot see: a source we have no full text for, which is not checked at all. A paywalled paper stays unchecked unless you upload it.

Every verdict is shown. All but one kind are marked experimental. A blind sample in October 2026 measured confirmed verdicts from the current checker on the sources a paper itself cites: 37 of 39 were right (95%). They show without a note. The other kinds — partly supported, disputed, misattributed and unverifiable — are not measured well enough yet, and no verdict on a claim written by you or by the assistant (a literature review’s claims, a claim you add), or on a paper you upload for a private check, is measured yet. So where one of those is shown with its label — on a claim’s row in the Library, in the chat’s citation panel and in literature review tables — it carries Experimental — accuracy not yet measured, and the public report’s count of checked sources ends with accuracy not yet measured while it counts one. Read the quote and the source before relying on a verdict. A check that decides a verdict counts against your daily claim checks; one that decides nothing doesn’t.

A check you ask for answers the same way. When you click Verify or Re-check on a claim you added about a paper in the Library, or ask the assistant to verify one of your claims, you see the verdict with the same note. Someone else’s claim, and a literature review’s claims copied into the Library, aren’t offered a check here.

References checked: N of M

A report says, for example, 12 of 30 cited sources checked. M counts every link from a claim to a reference it cites. N counts the links that were checked. The line then counts the sources that support the claim and those flagged, and ends with accuracy not yet measured while the verdicts are experimental.

M is often larger than N:

Background checks share a daily allowance of 200 across all papers, spread over the day.
They go in priority order: papers in someone’s Library, then claims shown first on a report, then papers someone opened or marked reviewed, then claims with no checked reference yet, then the oldest.
Some references can never be checked (they could not be resolved to a paper), and a broad citation with no specific claim is not checked. Both still count in M.
A reference whose source has no full text with us is not checked (see above). It counts in M, never in N.

An unchecked reference is never shown as a good one. Verify paper checks up to 8 of a paper’s references at once, the claims with findings first, without using your daily claim checks; when the paper is already verified, it runs the references queued for the background check right away. The rest follow from the background allowance.

Findings: numbers that don’t add up

Findings are machine-generated and can be wrong. Each one shows its quote and the numbers it recomputed, so you can check it yourself. On a paper in your Library you can Confirm or Dismiss each one.

Stats check recomputes each reported p-value from its test statistic and degrees of freedom (APA-style results such as t(28) = 2.20, p = .04). No model is involved.

It usually cannot tell whether the statistic or the p-value is the misprint. One exception: when an F test prints its partial eta squared, and that effect size fits the p-value but not the F, the finding says the F is the likely misprint and is marked minor.
It reads results written out in the text, including a full result such as t(28) = 2.20, p = .04 written inside a table cell. A result it cannot parse is skipped. Tables whose columns hold the statistic, its degrees of freedom and the p-value separately are checked row by row too, but those findings are not shown: in our tests too many of them were wrong.
Measured precision: in a blind sample of 199 findings across seven fields, 167 pointed at a real inconsistency (84%; 95% interval 78%–88%). In the three fields where it did worst, a fresh sample after our fixes scored 89% (neuroscience), 82% (psychology) and 84% (public health).

Checkup asks a model to find values that don’t add up: percentages, sums, differences, units, and text that contradicts itself. A finding is kept only when its quote appears in the paper and the server reproduces any arithmetic it rests on.

The server recomputes the numbers the model picked. It does not check that the model picked the right numbers from the paper. Most wrong findings are misreads of this kind.
Measured precision: arithmetic 38% (17 of 45; 95% interval 25%–52%), units 62% (24 of 39; 46%–75%). The arithmetic rule was tuned on the same sample. These samples were judged by AI models, not by people.
Contradictions are two passages of the paper that cannot both be true. Since October 2026, an independent AI check reads both passages in context and confirms or rejects each one. A contradiction is shown only when that check confirms it and agrees that its description names the actual conflict. Its major or minor rating also comes from that check.
Measured on a fresh sample of 200 contradictions, the ones the check confirmed were right 96% of the time (79 of 82; 95% interval 90%–99%), and it confirmed 91% of the real ones. Two AI models labelled the sample and a person decided where they disagreed; a person’s spot-check of 20 labels the two models agreed on confirmed all 20. Text whose licence lets us analyse it but not republish it was not part of that sample.

Every finding is shown only after an independent AI check has confirmed it: on the public report on a paper, in your Library, and on the private report on a PDF you upload. Until then it is counted as waiting for that check. The one exception is a finding in the Library that someone has already confirmed or dismissed: it keeps showing, with that review.

Checks we are still testing

Some checks run for our team only. They change nothing on your report:

A faster model that reads the cited passage and predicts the verdict. Its answers are stored for comparison and never used.

Text we analyse but don’t republish

Our analysis is ours; the paper’s text is not. Some copies of a paper come with a licence that allows text mining but not republishing: PubMed Central and OpenAlex copies without a Creative Commons licence. We analyse those copies, and we show short quotes from them as from any other copy:

the quote under a claim, an assumption or a finding;
the sentence where another paper cites the paper.

Claims, findings and the AI snapshot can use the paper’s own words. We don’t show the full text of such a copy as text. When there is no PDF to show, the Library’s PDF tab does not show this text; if the paper is in PubMed Central, the tab links to it. When the assistant reads such a paper in a chat, it gets only the abstract, and the PubMed Central link if there is one.

What text we read

We read the full text where we can find an open copy or you upload one. Otherwise we read the abstract.

The Full text cell in the Library shows what CoChat holds: indexed, PDF, paywalled, abstract only, or none. It is not a record of what one check read.
Stats check says Only the abstract was available when that is what it read.
A public report whose paper has no open full text says so, and runs no claims and no self-check.
A snapshot is written from the full text only.

Limits that apply to every report

Paywalls. A closed paper is read from its abstract. Most checks need the full text. In our first test against seven trials with known problems, only two had open text we could screen.
Tables. Tables are kept from PubMed Central and Europe PMC text, and from PDFs our layout parser reads. A PDF read by the fallback parser, or a publisher web page, can lose its table layout, and the checks then miss numbers that sit only in a table.
Models. Extraction, checkup, snapshot, claim verification and claim labels each run on a model we configure, and the configured model can change. A public report names the model that read its claims.
Coverage. We have not yet measured how many problem papers these checks catch, or how often they flag a sound one.

Found a mistake on a report? Use Request a review at the bottom of it.