Citing Less Critically

An EMNLP 2026 paper asks whether large language models cite the way scholars do. I designed two of its figures, built the pipeline that resolved its citations to bibliographic records, and served as one of three human annotators on its validation set.

The problem

The study needed three things that are easy to state and hard to build: a way to compare human and model citation behaviour at the same position in the same sentence, a way to label what a citation is rhetorically doing, and a way to look up what each cited paper actually is.

I worked on the third, and on explaining the first.

The lookup problem is where the paper’s central inference lives. Across 1,746 conference papers there are 132,913 citation slots, each naming a work that has to be resolved to a real record before anything can be said about how old it is, how cited it is, or who wrote it. Models produce references that are sometimes real, sometimes real but garbled, and sometimes invented. If the matching layer is inconsistent, every downstream number is unreadable.

Approach

A four-tier match cascade, ordered by cost. DOI lookup first, then arXiv DOI (10.48550/arXiv.<ID>), then PMID (skipped, the data had none), then fuzzy title-and-year search. Cheap and exact before expensive and probabilistic. Most references never reach the last tier.

A similarity threshold instead of taking the top hit. A bibliographic search endpoint will happily return a related but different paper. Accepting the first result silently injects wrong matches that look like successful ones. Normalised title comparison with a 0.85 floor means the pipeline drops a borderline case rather than guessing at it. Under-matching is recoverable; a wrong match is not, because nothing downstream will flag it.

A recorded match_method on every row. The output says whether a citation resolved by DOI, by arXiv ID, or by title. That turns a single aggregate match rate into something auditable by tier, which matters when a reviewer asks how much of the matching rests on fuzzy comparison.

Deduplicate before requesting. The same cited work appears many times across tens of thousands of citations. Keying on DOI, falling back to normalised arXiv ID, falling back to normalised title, collapses those into one request each. The script reports how many API calls that saved.

Sample mode, then full, with an on-disk HTTP cache. Validate on fifty rows before spending hours of requests, and make the full re-run free so that fixing one line does not mean re-fetching everything.

Non-destructive corrections. Fifteen citing papers had truncated arXiv identifiers, found by cross-comparing three datasets. The fix script looks each one up by conference DOI with three fallback paths, then falls back again to a title search, and writes a mapping table rather than editing the source files. It also asserts that each corrected identifier starts with the truncated one it replaces, so a bad lookup fails loudly instead of overwriting good data.

What I built

The matching pipeline. Human citations resolve at 86.7 percent. The six models resolve at between 39.5 and 81.9 percent.

That spread is the paper’s evidence for hallucination, and it is only usable because both sides go through the same pipeline. Since human references match at a high rate, the gap is attributable to what the models produced rather than to how the matching worked. An audit of a hundred unmatched titles per model then puts the hallucination share at 79 percent for one model and 97 percent for another. None of that reads as a finding if the matching layer is doing something different for the two sides.

Figure 1, the framework diagram. Three stages, one citation slot travelling through all of them: the human original, the masked version, the model’s replacement. The difficulty is not drawing boxes. It is that stages run in sequence while six models run in parallel inside stage one, and both have to be legible in one static frame. The three versions of the sentence sit on a shared horizontal line so a reader can compare them directly, with the stage boundaries carrying the sequence.

Figure 4a, the distance schematic. The paper measures social distance as the mean shortest path across four author-role pairs between the citing and cited papers. As a formula nobody reads it. As a picture of two named papers with four paths drawn between their first and last authors, and the four numbers averaged, a reader sees where a value of 2.25 comes from. It is the only purely explanatory figure in the paper; everything else plots results.

The annotation rubric, and one of three annotation passes. The three intent definitions the whole study runs on, supporting, contrasting and mentioning, were settled by the first three authors together, adapted from an existing citation-classification scheme. I then labelled a stratified sample of ninety citation sentences blind to the model labels and to the cited papers, as one of three annotators.

That validation set is load-bearing. It answers the most obvious objection to the paper, which is that an LLM judge might simply read LLM-written prose as warmer. The warming appears in all three humans’ labels too, in the same direction, so it is a property of the rewritten sentences rather than a judge preference.

Rebuilding Figure 1 under review

The submitted version of Figure 1 came back from review with a note that the formatting was generally solid but the figure was too visually dense. I rebuilt it, and the rebuild is the figure in the published paper.

Density in a diagram is rarely a matter of too many elements. It is usually too few levels: everything competing at the same weight, so a reader has nowhere to start. That is why I restarted rather than adjusted. Hierarchy is the lowest-level decision in a layout, and local edits to an existing composition cannot move it.

The rebuild made the three stages explicit horizontal bands, dropped the parallel branches inside stage one to a secondary weight, and made the shared horizontal line comparing the three versions of one sentence the first thing a reader lands on. Nothing was removed from the method. Six further rounds of team discussion refined it to camera-ready.

Every frame is still on the Figma canvas, before and after, from plot_diagram 2 through plot_diagram_update and then six frames of plot_diagram_better.

The reason this is worth more than the final figure alone: the critique came from an anonymous expert reviewing the work for publication, the response is traceable frame by frame, and the result cleared a main-conference review. Self-directed iteration is easy to claim. This is the other kind.

Figure 4a took one pass and one round of minor type-size adjustment. Put next to Figure 1 that contrast says the useful thing: the difficulty is not in the drawing, it is in how many levels of structure the subject has. One abstract measure needs one worked example. A three-stage pipeline with parallel branches inside one stage needs its hierarchy sorted out before it can be read at all.

Key decisions

Under-match rather than mis-match. Everything about the pipeline follows from this. The threshold, the cost-ordered cascade, the recorded method, the assertion in the correction script. A missing match shows up as a lower rate and gets discussed. A wrong match is invisible and contaminates a published number.

Explain one abstraction rather than illustrate the whole method. Figure 4a exists because the coauthorship-distance metric is the one definition in the paper a reader is most likely to skip. Spending a figure on a single idea beats spreading it thin.

Pipeline notes

Python with pandas and requests, using an HTTPAdapter with urllib3 retry. Request pacing at 0.11 seconds, roughly nine requests per second against a polite-pool ceiling of ten, with four worker threads sharing a global rate cap. Responses cached to JSON on disk.

The uploaded scripts are the OpenAlex version, an earlier attempt at 200-paper scale with roughly 14,000 real and 9,500 model-generated references. The published results use Dimensions at the full 1,746-paper scale; the matching strategy is the same.

Match reporting is per-tier, so the contribution of exact versus fuzzy matching is visible rather than pooled.

Outcome

Accepted at EMNLP 2026, main conference. Preprint and code are public.

I am the third of five authors, and Human annotator 3 in the paper’s validation table. The equal-contribution marks belong to the first two. The research question, the masked-citation design, the judging procedure and the 20.3-million-edge coauthorship analysis are not mine, and I would rather say so than let a reader assume otherwise. What is mine is above.

Reflection

The useful thing I learned is that in a pipeline feeding a published claim, the failure modes are not symmetric. Recall problems announce themselves as a number somebody will ask about. Precision problems do not announce themselves at all: a wrong match becomes a real-looking row in a real-looking table, and the person who eventually notices is a reader, after publication. Designing for the loud failure over the quiet one is not caution, it is the only version that survives review.

The figures taught me something narrower. Reviewers of a methods paper read for whether the method is sound, not for whether the diagram is handsome, and a framework figure earns its space by making one thing checkable. The density note was the most useful piece of design feedback I have had, because it came from someone with no stake in the diagram and every stake in understanding the method.

Figure 4a is the one I would show if asked for a single example of information design: a definition nobody would read, turned into something a reader gets in one pass. It needed one revision. Figure 1 needed nine, which is the more honest number.