Citing Less Critically

Redesigning the framework figure

Our team studied whether LLMs preserve human citation choices and intent when reconstructing a masked citation sentence. In this controlled task, the models produced fewer contrasting citations than the human-written baseline.

I designed the final framework figure to explain how the study reconstructs citation sentences, classifies their intent, and matches references to bibliographic records. Blue and red labels identify the human and LLM-generated material throughout the three stages.

Earlier version and redesign

The earlier version below was created by Yixuan Liu. A reviewer described it as “too visually dense.” I rebuilt the figure through six rounds of team feedback, producing the final version shown above for the paper’s final manuscript.

I organized the method into three horizontal bands, giving each stage a heading and a consistent top-to-bottom reading order. Within the labeling stage, I kept the original and generated sentences on separate, parallel paths so readers could follow both through the same judging procedure.

I also reduced the emphasis on surrounding text while retaining the citation sentences and reference lists needed to understand the example. In the final stage, I condensed the branching analysis details into a single row linking matched records, metadata, and the three research questions.

Framework iteration

My first substantial redesign separated the method into three stages, with intent labeling and bibliographic matching placed side by side beneath reconstruction. I later stacked all three stages vertically and simplified the final analysis row, keeping the reading direction consistent.

The final figure retains a worked citation example alongside the method. This preserves the detail needed to inspect the experiment, although the sentence-level text still requires enlargement at smaller display sizes.

Explaining social distance

The study also compares how closely citing and cited authors are connected through coauthorship. I designed Figure 4a to show how the distance measure is calculated.

I arranged each example from left to right: identify the first and last authors, trace the four author-pair paths through the network, and then average their lengths. Repeating this structure for an original citation and an LLM-generated replacement lets readers compare the calculation step by step.

Distance calculation

The first and last authors of the citing paper are paired with the first and last authors of the cited paper, giving four author pairs. The measure averages the shortest coauthorship-path lengths for reachable author pairs. Pairs in different connected components are marked unreachable, and pairs with missing researcher identifiers are excluded.

The diagram explains this calculation. The network analysis was conducted by the research team; my contribution was its visual explanation. Coauthorship distance describes connections in the recorded network and does not directly measure personal familiarity or favoritism.

Matching citation records

I built the pipeline that linked human and LLM-generated references to Dimensions records for downstream analysis. It matched references by DOI first, then used normalized titles when DOI matching failed.

For the human baseline, 115,278 of 132,913 citation slots matched a Dimensions record, a coverage rate of 86.7%. Each slot represents one cited work at a citation position, so repeated citations to the same paper are counted separately. Unmatched references were excluded from downstream analysis.

I also participated in reviewing 200 unmatched titles and calibrating the pipeline. The review distinguished missed matches, real works with inaccurate titles, and fabricated references, helping separate matching failures from problems in the generated citations.

Matching and unmatched references

The title fallback used exact matching after normalization. Successful matches supplied canonical metadata from Dimensions; author names were not used as matching keys because of formatting variation. The reported match rate measures coverage, not the accuracy of every matched record.

For the unmatched-reference review reported in Appendix E, the team sampled 100 titles from GPT-5.1 and 100 from Claude-3.5-Haiku. I participated in the human review and pipeline calibration; the audit also used Claude Opus 4.8 with web search.

The review separated three outcomes: a real record the pipeline had missed, a real work with a substantially inaccurate title, and a fabricated reference. The paper groups the latter two as hallucinations. These categories describe the sampled unmatched titles, not all citations generated by either model.

The downstream findings therefore describe matched references and may not represent the unmatched outputs.

Checking citation intent

The study used an LLM judge to classify citation intent. I helped refine the annotation definitions and served as one of three independent human annotators, labeling citations as supporting, contrasting, or mentioning. We worked without seeing the automated labels or cited-paper identities.

In the 77 paired examples evaluated, each human annotator labeled a smaller share of GPT-5.1’s reconstructed sentences as contrasting than of the originals. This supported the direction of the automated finding within this sample.

Human validation

The validation set contained 90 original citation sentences, stratified across the three intent categories. On these sentences, the primary LLM judge agreed with the human-majority label in 73% of cases, with Cohen’s κ = 0.60.

The before-and-after comparison used the 77 of those 90 positions for which GPT-5.1 returned a citation sentence. Each human annotator classified a smaller share of the reconstructed sentences as contrasting than of the originals.

These are two related checks with different denominators: agreement on the 90 original sentences, and change in intent across 77 original–reconstructed pairs. The sample does not establish labeling accuracy across the full corpus or every model.