OJR vs SNIP and SJR: journal ranking validation

Key findings

• OJR and CWTS SNIP rank journals in a broadly similar order. On 18,257 journals present in both sources, ρ between OJR and SNIP is 0.789 (95% interval 0.782–0.795) for 2024 and 0.802 (0.796–0.809) for 2025. Still, 16% of journals differ by more than 25 percentiles between the two (15% for 2025). For comparison, the literature reports ρ = 0.95 between two field-normalized indicators that differ only in how they average (ratio of sums against the mean normalized citation score, 8,423 journals)^[1], and ρ = 0.645 between SNIP and the raw journal impact factor (9,657 journals)^[2].
• OJR and CWTS SNIP agree most closely inside a subfield. Within the 145 OJR subfields with at least 20 matched journals, the median ρ is 0.832, and 86% of subfields reach 0.70 or more.
• Agreement between OJR and CWTS SNIP varies by field, from ρ 0.63 to 0.93. Arts and Humanities is lowest (0.634, 0.569–0.693), then Decision Sciences and Mathematics. Pharmacology, Chemistry and Environmental Science are highest.
• OJR and SNIP quartiles usually agree to within one step. The published OJR quartile matches a SNIP quartile computed in the same subfield for 56% of journals, and is within one quartile for 95% (linear weighted κ 0.60). When both quartiles are computed on the same matched journals, exact agreement is 61% (κ 0.64).
• Where OJR and CWTS SNIP disagree, the reasons are partly identifiable; each driver is weak on its own (|ρ| ≤ 0.20). Journals that rank much higher on SNIP more often print meeting abstracts that OpenAlex types as articles: in the typical such journal, 6.3% of papers have no matched reference, against 1.3% elsewhere (medians), and a minority have far more, which lifts the group mean to 25.1% against 9.4%. Mathematics and computer science journals are over-represented among those ranking much higher on SNIP. Journals that rank much higher on OJR are mostly small, and mostly in the social sciences; some conference-proceedings series are among them.
• Leaving out meeting abstracts brings OJR closer to CWTS SNIP, most of all in Health Sciences. For 2024, the abstract rule applied from release 2025 raises ρ with SNIP by 0.017 overall (0.015–0.019) and by 0.057 in Health Sciences (0.047–0.066). A stricter rule that drops every paper without a matched reference gives 0.027 and 0.060. Because Scopus already leaves most meeting abstracts out of its citable types, part of this gain is expected by construction. Release 2025 introduced this rule; this release applies it.
• SNIP predicts the Norwegian expert levels better than OJR does. Separating Level 2 from Level 1 journals within the register’s fields, on the 14,901 journals with SNIP and every rule score, SNIP reaches an AUC of 0.841 and OJR 0.785 (gap 0.056, 95% interval 0.047–0.065). SNIP is ahead in 52 of the 56 register fields tested. By OJR field, the gap is widest in Economics, Decision Sciences, Medicine and Computer Science. About half of it is already present in raw citation counts. The abstract rule applied from release 2025 lifts OJR to 0.793; we have not shown that OJR will close the gap.
• OJR and SCImago’s SJR also rank journals in a broadly similar order, but SNIP and SJR agree with each other more closely than either agrees with OJR. For 2024, ρ is 0.782 (0.775–0.789) between OJR and SJR on 18,034 journals, against 0.850 (0.845–0.855) between SNIP and SJR on the 17,943 journals with all three. On those 17,943 journals the median ρ within subfields is 0.856 between OJR and SJR, 0.829 between OJR and SNIP, and 0.893 between SNIP and SJR.
• Within the register’s fields, SCImago’s SJR predicts the Norwegian expert levels best of OJR, CWTS SNIP and SJR. On the 14,775 journals with all three scores, SJR reaches a within-field AUC of 0.868 (0.859–0.878) for 2024, SNIP 0.841 and OJR 0.786. SJR is ahead of OJR by 0.083 (0.074–0.092), and in every one of the 21 OJR fields tested. Pooled across fields, SNIP is ahead (0.835 against 0.825 for SJR).
• CWTS SNIP and SJR are not ground truth, and agreement with them is not proof of validity. They are Scopus-based indicators with their own design choices and coverage, and they share a database, which helps them agree with each other. Agreement shows that OJR measures something close to what established indicators measure; it does not show that any of them measures journal quality. See what OJR is not and our responsible-use policy.

How we did this

Why compare with SNIP

No ranking of journals can be checked against a true answer. What can be checked is whether an indicator behaves like the established ones that measure the same thing, and whether it differs for reasons that can be explained. SNIP is the closest established relative of OJR^{[6, 7]}:

• Both are size-independent, field-normalized measures of citation impact.
• Both count citations in year Y to papers published from Y−3 to Y−1.
• Both count only articles, reviews and conference papers.

They differ in four ways, and each one should produce some disagreement:

• Database. SNIP uses Scopus; OJR uses OpenAlex. The two index different journals and type documents differently.
• Normalization. SNIP normalizes on the citing side, by the reference lists of the papers that cite a journal. OJR normalizes on the cited side, by subfield, year and document type. OJR therefore treats reviews as their own type; SNIP does not.
• Self-citations. SNIP and IPP include a journal’s citations to itself; OJR excludes them.
• Citing sources. CWTS ignores citations from trade journals and from sources with few references to other covered sources. OJR counts every citing journal in OpenAlex but excludes repositories.

Our earlier literature review expected OJR to correlate with SNIP at about 0.65–0.85, and noted that no study had yet validated an OpenAlex-based journal indicator against Scopus metrics.

Data and method

Sources

• OJR. Release 2026-Q4.1, computed from the OpenAlex snapshot of 2026-09-23. The release covers 33,339 CWTS Core journals, of which about 21,500–22,000 are ranked in each recent year.
• CWTS Journal Indicators. The March 2026 release file (39,016 Scopus sources, years 1999–2025). Source: CWTS Journal Indicators, Centre for Science and Technology Studies (CWTS), Leiden University, based on Scopus data from Elsevier. The CWTS site gives no specific attribution wording, so we follow its copyright notice. This article contains only aggregate statistics: no SNIP, IPP or paper count for any single journal.
• Norwegian Register. The journal export of the Norwegian Register for Scientific Journals, Series and Publishers, operated by the Directorate for Higher Education and Skills (HK-dir), taken in October 2026 (40,244 channels, of which 3,974 have no level recorded). Used under CC BY 4.0^[8].
• SCImago Journal Rank. SCImago’s yearly exports for 1999–2025 (768,391 source-years), taken from the sjrdata R package mirror (version 1.0.25, June 2026)^[9] of the files SCImago publishes^[10]. SCImago’s site says its information may be used for non-commercial purposes with a citation. As with SNIP, only aggregate statistics are published: no SJR value for any single journal.
• Abstract-rule scores. Per-journal OJR scores for 2024 and 2025 under the abstract rule (R8+clin) and the earlier candidate R5, computed on the same snapshot for the change-log entry. They are research scores, not published ones; their copy of the published OJR matches the release exactly.

Matching

ISSNs were normalized to the form NNNN-NNNC with an uppercase X; malformed values were ignored. An OJR journal’s ISSN-L and every listed ISSN were matched against the print and electronic ISSNs of the CWTS sources for the same year.

• Ambiguity was rare. In 2024, 9 journals matched more than one CWTS source. One was settled by the number of shared ISSNs and eight by title similarity.
• Two CWTS sources were each claimed by two OJR journals. The closer match kept the source, and the other journal was dropped from that year.
• 64.6% of all 33,339 core journals share an ISSN with some CWTS source in some year. Of ranked journals, 83.5% have a SNIP for 2024.

Years

The CWTS methodology defines IPP as citations in the present year to the publications of the three preceding years^[11]. That is the same window OJR uses (citation windows), so OJR year Y is compared with CWTS year Y. We confirmed this in the data rather than relying on the label:

• Paper counts. CWTS paper counts (P) for year 2024 are within 10% of OJR’s citable-paper counts for 55% of journals. Against CWTS 2023 and 2025 the figure is 34% and 40%.
• Raw citations per paper. On the 18,077 journals matched in all three years, OpenAlex’s raw citations per paper correlate most strongly with IPP of the same year label: 0.944, against 0.906 and 0.935 for the neighbouring years.

The main years are 2025, 2024 and 2023. For stability, 2022, 2021, 2020, 2015 and 2010 are also reported.

Statistics

• Population. Ranked OJR journals with a CWTS value for the same year.
• Correlations. Spearman’s ρ with bootstrap 95% intervals (1,000 resamples of journals, fixed seed), overall, by domain and by field. Fields with fewer than 30 matched journals are not shown.
• Within-subfield agreement. ρ inside each OJR subfield with at least 20 matched journals.
• Quartiles. The published OJR subfield quartile against a SNIP quartile computed by the same rule inside the same subfield on matched journals. Subfields need at least 10 matched journals.
• Disagreement profile. Each journal’s percentile rank on SNIP minus its percentile rank on OJR. The 5% with the largest difference in each direction are compared with the other 90%.
• Larger journals. Everything is repeated on journals with at least 100 papers in both sources, to separate small-journal noise.
• Same journals for every score compared. When several scores are compared, only journals that have all of them are used, so the same quantity can differ slightly between tables. For 2024: OJR against SNIP, 18,257 journals; the abstract rules against SNIP, 18,170 (journals with every rule score); OJR, SNIP and SJR together, 17,943; the abstract rules against SJR, a different set of 17,943 (journals with an SJR value and every rule score); the Norwegian levels with SNIP, IPP and the rule scores, 14,901, and with SJR as well, 14,775. Each table names its population.

Results

Overall agreement

Agreement between OJR and SNIP is fairly steady over 15 years (ρ 0.75 to 0.80), and slightly higher in recent years. Agreement between OJR and IPP changes more: it falls from 0.79 for 2025 to 0.67 for 2010. Two further comparisons help to read the table:

• SNIP and IPP agree at about 0.80, the same level as OJR and SNIP. This is not a like-for-like benchmark: it compares a normalized indicator with its own raw counts in the same database. It shows only that field normalization alone moves journals around a good deal.
• The two databases agree closely on raw counts. On all 18,257 matched journals, OpenAlex’s raw citations per paper (self-citations included, as in IPP) correlate with IPP at 0.943 (0.940–0.946) for 2024. Health Sciences is the lowest domain, at 0.901. So most of the remaining difference comes from how the indicators are built, not from the databases.

On the 12,040 larger journals (at least 100 papers in both sources), ρ rises to 0.812 (0.804–0.820) for 2024 and 0.824 (0.816–0.831) for 2025. The within-subfield median hardly changes (0.831 and 0.844). Small journals add noise but do not explain most of the disagreement.

By domain and field (2024)

Chemical Engineering has too few matched journals (under 30) to report. The 2025 pattern is the same, with ρ from 0.649 (Arts and Humanities) to 0.901 (Environmental Science). Match rates by field range from 77–78% (Pharmacology, Computer Science, Health Professions, Nursing) to 92–95% (Mathematics, Earth and Planetary Sciences). Journals OJR ranks but Scopus does not index are outside this test.

Within subfields

A field-normalized indicator is mainly used to compare journals in the same subfield, so this is the comparison that matters most to users.

• 2024: median ρ 0.832 (interquartile range 0.766–0.871) across 145 subfields; 86% of subfields at 0.70 or above; none below 0.50.
• 2025: median 0.838 across 147 subfields; 90% at 0.70 or above.

The lowest subfields are mostly small humanities and theory subfields: Computational Theory and Mathematics, History and Philosophy of Science, Law, Philosophy, Conservation. Their citation counts are low, and the two databases cover their journals differently. In 2025 one of them, History and Philosophy of Science (24 journals), shows no relation at all.

Quartile agreement (2024)

The published comparison is not symmetric. Matched journals rank higher on OJR than the unmatched ones, so OJR’s Q1 and Q2 rows are fuller than its Q4 row. Recomputing both quartiles on the same journals removes that tilt, and is the fairer measure of agreement.

Fewer than 0.5% of journals sit in opposite extremes (Q1 on one indicator, Q4 on the other). A one-quartile difference, by contrast, is common: about 38% of journals. It should not be read as a real difference in impact; many journals sit near a quartile boundary.

Where and why they disagree

For each journal we took the difference between its percentile rank on SNIP and on OJR. In 2024, 16% of matched journals differ by more than 25 percentiles. The 5% with the largest gap in each direction differ by at least 33 (SNIP higher) or 30 (OJR higher) percentiles.

The zero-reference share is very skewed. The typical journal in every group has few zero-reference papers; the higher mean in the SNIP-higher group comes from a minority of journals where most papers have none.

Across all matched journals, the rank gap (SNIP minus OJR) correlates with self-citation share (ρ 0.20), journal size (0.17) and zero-reference share (0.15). It correlates negatively with the share of citations from low-coverage publishers (−0.19). Each driver is weak on its own, and together they account for only part of the large gaps.

The reasons, as far as the data show:

• Meeting abstracts typed as articles. Large clinical journals that print congress abstracts in supplements are common among those ranking much higher on SNIP; in some of them nearly all of OpenAlex’s citable papers have no matched reference. OpenAlex types these abstracts as articles, which inflates OJR’s expected citations. Scopus appears to leave most of them out of its citable types: for the SNIP-higher group, Scopus counts a median 15% fewer papers than OpenAlex, against 5% fewer for the rest.
• Unusual OpenAlex source records. A few journals ranking much higher on SNIP have OpenAlex records that hold thousands of works without references, or are placed in a subfield far from their subject. For them the difference comes from the record, not from the method.
• Self-citation. SNIP counts self-citations and OJR does not (which citations count), so journals that cite themselves more rank higher on SNIP. SNIP-higher journals have a median self-citation share of 5.5%, against 4.0% for the rest.
• Field normalization in mathematics and computer science. Mathematics journals are 4.4 times over-represented among those ranking much higher on SNIP, and computer science 2.4 times (table above). SNIP’s citing-side normalization credits fields whose papers cite few recent works. OJR’s subfield baselines handle these fields differently. Differences in how the two databases cover computer-science conference papers may add to this; we have not tested it.
• Small social-science journals cited from outside Scopus. Journals ranking much higher on OJR are mostly small (59% under 100 papers), and 76% are in the Social Sciences domain; business, general social science and economics are the most over-represented fields (table above). A larger share of their citations comes from publishers whose reference lists OpenAlex holds only in part, which are often small and regional. Many such citing journals are not in Scopus at all, or not counted by CWTS as citing sources, so these citations count for OJR but not for SNIP. This is our reading, not a tested result.
• Conference-proceedings series. Some conference-proceedings series rank much higher on OJR than on SNIP: 2.4% of that group are CWTS proceedings sources, against 0.2% of the other 90%. OpenAlex treats these series as journals and normalizes them against journal papers of the same subfield and type; Scopus classifies most of them as proceedings.
• Reviews. Review-heavy journals rank somewhat higher on SNIP: the mean review share is 6.4% in that group against 2.0% elsewhere. That is expected, because OJR compares reviews with reviews and SNIP does not. The rank correlation with the gap is weak and negative (−0.14), however, so review share is not a clean driver once fields are mixed.

Expert judgement: the Norwegian Register

The Norwegian Register classifies publication channels into levels: 0 (not approved as scholarly), 1 (scholarly), and 2 (leading channels). Field panels choose the Level 2 journals, and Level 2 is capped at about 20% of each field’s publications^[12]. That makes Level 2 a judgement made within fields. Earlier work found that SNIP agreed with the Norwegian levels better than raw citation counts did^[13].

Matching

• 20,780 of the 33,339 core journals (62.3%) are in the register.
• For 2024, 16,862 of 21,853 ranked journals (77.2%) are matched: 1,745 at Level 2, 13,882 at Level 1, 1,089 at Level 0, 14 under discussion (X) and 132 with no level recorded.
• In 232 cases more than one register entry matched a journal. 90 were settled by shared ISSNs and 133 by title. In the other 9 (8 for 2025) the entries were identical on both, and the register has no paper counts to compare, so the first entry in the register’s order was kept.

OJR by level (2024)

OJR orders the levels as expected. Separating Level 1 and 2 journals from Level 0 gives an AUC of 0.746 (0.729–0.760).

Level 2 against Level 1 (2024)

“Within the register’s fields” compares Level 2 and Level 1 journals only inside the same register field. That respects the 20% quota, and it is the fairer test.

• SNIP predicts Level 2 better than OJR does. The gap within fields is 0.056 (0.047–0.065), and it is the same for 2025 (0.836 against 0.781). Across the 56 register fields with at least five journals at each level, the median within-field AUC is 0.788 for OJR and 0.864 for SNIP. SNIP is ahead in 52 of them and OJR in 4.
• About half of the gap is in the data, before any normalization. Scopus raw citations per paper (IPP) beat OpenAlex raw citations per paper by 0.031 within fields. OJR’s normalization adds 0.016 to its raw counts; SNIP’s adds 0.041 to its own.
• By OJR field, the gap is widest in Economics, Decision Sciences, Medicine and Computer Science. Within those fields, SNIP’s AUC exceeds OJR’s by 0.178 in Economics, Econometrics and Finance, 0.169 in Decision Sciences, 0.120 in Medicine and 0.119 in Computer Science. Health Sciences has the largest gap of the four domains (0.106).
• The abstract rule explains most of the Medicine gap, but not the others. In Medicine OJR’s AUC is 0.768, IPP’s 0.884 and SNIP’s 0.888; the abstract rule lifts OJR to 0.854 (the Leiden rule to 0.855). In Economics, Decision Sciences and Computer Science the abstract rule changes OJR’s AUC by 0.002 or less, because its publisher-coverage test applies only in Health Sciences.
• By OJR field, OJR’s AUC runs from 0.657 (Arts and Humanities) to 0.872 (Chemistry). SNIP is ahead in 20 of the 21 fields with at least 10 journals at each level; OJR is ahead in Dentistry.

The register’s panels decide levels with their own knowledge of the journals, and they may also look at established citation metrics, including Scopus-based ones. If they do, the comparison favours SNIP to some degree. We have not measured how much.

OJR vs SJR, and the SNIP vs SJR correlation

SNIP is one indicator. SJR, computed by SCImago from the same Scopus data, is the other widely used Scopus journal indicator, and it is built differently from both SNIP and OJR^[3]. It weights citations by the prestige of the citing journal, shared out over that journal’s references; that adjusts partly for how much a field cites, but SJR has no explicit field or subfield baseline as OJR and SNIP do. Comparing OJR with SJR, and SNIP with SJR, shows how much of the agreement above is specific to SNIP. The two Scopus indicators correlate with each other at ρ 0.850 for 2024 and 0.863 for 2025, higher than either correlates with OJR.

Matching and years

The same ISSN rule matches 18,034 of 21,853 ranked OJR journals (82.5%) to an SJR value for 2024, and 18,166 (84.2%) for 2025; 66.7% of all core journals appear in SCImago in some year. SJR for year Y counts citations in Y to documents of Y−3 to Y−1, the same window as OJR and SNIP. The data confirm the label: SCImago’s citable documents for 2024 are within 10% of OJR’s citable papers for 56% of journals, against 35% and 41% for 2023 and 2025, and raw citations per document correlate most strongly for the same label (0.948, against 0.899 and 0.934).

Agreement (2024)

For 2025, on the 18,125 journals with all three, ρ is 0.790 (0.783–0.797) between OJR and SJR and 0.863 (0.858–0.868) between SNIP and SJR; the within-subfield medians are 0.867 and 0.911.

• Overall, OJR agrees with SJR about as well as with SNIP. SNIP and SJR, which share Scopus, agree more closely with each other.
• Within subfields, OJR agrees better with SJR than with SNIP (median 0.856 against 0.829 for 2024, same journals). Once journals are compared inside one subfield, SJR’s lack of an explicit field baseline matters less.
• By domain (2024), ρ between OJR and SJR is 0.853 in the Life Sciences, 0.847 in the Physical Sciences, 0.833 in Health Sciences and 0.781 in the Social Sciences.
• By field, the pattern seen with SNIP repeats. ρ is lowest in Decision Sciences (0.678, 0.567–0.763), Mathematics (0.711), Arts and Humanities (0.730) and Economics, Econometrics and Finance (0.740), and highest in Pharmacology, Toxicology and Pharmaceutics (0.953) and Environmental Science (0.918). In Decision Sciences, Computer Science and Economics, SNIP and SJR agree at 0.89–0.92, so OJR is the outlier there.

Quartiles (2024)

SCImago assigns each journal to its own categories, which use the same subject names as OJR’s subfields. Only 42% of matched journals have a SCImago category with the name of their OJR subfield, so the first row covers a minority of journals. The recomputed comparison covers almost all of them.

Norwegian expert levels (2024)

Within the register’s fields SJR does best, ahead of OJR by 0.083 (0.074–0.092). It is ahead of OJR in all 21 OJR fields with at least 10 journals at each level, including Medicine (0.905 against 0.767) and Economics, Econometrics and Finance (0.860 against 0.685). For 2025 the picture is the same: SJR 0.859, SNIP 0.836, OJR 0.781. Pooled across fields, SNIP is slightly ahead of SJR, as expected for a field-normalized indicator against a quota set within fields. Earlier work found that SNIP and SJR both track the Norwegian levels better than raw citation counts^[13]; here both also beat the raw counts of the earlier Norwegian table (0.769 for OpenAlex and 0.800 for Scopus, within fields), and OJR is behind both.

The abstract rules against SJR (2024)

On the 17,943 journals with an SJR value and every rule score, where OJR’s ρ with SJR is 0.780, the abstract rule raises ρ with SJR from 0.780 to 0.793 (+0.012, 0.010–0.015). In Health Sciences it rises from 0.833 to 0.888 (+0.055, 0.045–0.065), and the median within-subfield ρ there from 0.832 to 0.902. The stricter Leiden rule raises the overall figure much more, to 0.821 (+0.041, 0.037–0.045). The papers it removes beyond the abstract rule belong to publishers that deposit few references, and many of them are research articles, so this extra gain is not evidence that they are front matter; we have not established its cause. Quartile κ with SJR rises from 0.675 to 0.690 overall and from 0.674 to 0.724 in Health Sciences under the abstract rule. On the Norwegian levels the abstract rule lifts OJR from 0.786 to 0.794 within fields, still well below SJR.

The same circularity applies as with SNIP. SJR is computed from Scopus, which leaves most meeting abstracts out of its citable types, so better agreement after removing them is partly expected by construction.

What this means for users

• Within a subfield, OJR and SNIP mostly tell the same story. If two journals in one subfield are far apart on OJR, they are very likely far apart on SNIP too.
• Treat a one-quartile difference between OJR and another ranking as normal. It happens for about 38% of journals. A two-quartile difference happens for about 5% and is worth a look.
• Be careful with large clinical journals that print congress abstracts. Releases before 2025 understate many of them, because OpenAlex counts their abstracts as articles; this release applies the abstract rule, which removes most of these papers (see below).
• Be careful with small journals and with conference-proceedings series. Small journals move a lot between indicators. Some conference-proceedings series rank higher on OJR than on SNIP.
• In economics, decision sciences and computer science, OJR agrees less with both SNIP and SJR (ρ 0.68–0.77, against 0.89–0.92 between SNIP and SJR). This study does not show which way individual journals differ. Arts and humanities rankings agree least of all, on any indicator.
• For decisions that mirror expert panels, OJR is currently the weakest of the three indicators tested. On the Norwegian levels within fields SJR does best, then SNIP, then OJR. OJR describes the citation impact of journals, not the quality of individual articles or people; see our responsible-use policy.

Limitations

• The expert levels are a second check, not an independent truth. They are one country’s judgement, and the panels may draw on citation metrics themselves.
• The test covers only journals in Scopus. That is 83.5% of OJR’s ranked journals for 2024. Journals that only OpenAlex covers are often regional or non-English, and they may behave differently. This study says nothing about them.
• The indicators differ by design. Self-citations, citing-source rules, document typing and the kind of normalization all differ. A correlation near 1 is neither expected nor desirable.
• Bootstrap intervals cover only sampling variation among journals. They do not include the uncertainty of each journal’s own score.
• The disagreement profile is descriptive. Field, size and document mix are tangled together, and the drivers above are associations, not causal estimates.
• The quartile comparison mixes populations. The published comparison uses OJR quartiles from all ranked journals and SNIP quartiles from matched journals only; the symmetric version corrects this.
• Data versions differ. The CWTS file is the March 2026 release; the OpenAlex snapshot is of 2026-09-23. Citations in 2025 may be slightly less complete in the CWTS file. The year alignment is confirmed by paper counts, but OJR 2024 correlates very slightly better with SNIP 2025 than with SNIP 2024 (0.793 against 0.790).
• Field labels come from OpenAlex. By-field results use OJR’s subfields. The register’s fields are identified only by number in the export, so they are not named here.
• The abstract-exclusion scores come from separate analyses. They are research variants computed on the same snapshot, not published scores.
• The analysis order cannot be checked. The plan was written down only after the full run, and the SJR comparison was added after the SNIP results were known.
• The SJR data come through a mirror. We used the sjrdata package’s copy of SCImago’s yearly files, not files downloaded from SCImago directly. Its year labels match SCImago’s on paper counts and raw citations (above), but we did not compare it value by value with SCImago’s site.
• SJR quartiles are per SCImago category. The published-quartile comparison covers only the 42% of journals whose SCImago categories include their OJR subfield’s name.

The abstract-exclusion check

OpenAlex types many meeting abstracts as articles (document types). We recomputed OJR under several rules:

• The Leiden rule, the front-matter filter of the Leiden Ranking Open Edition^[14]: only citable papers with at least one matched reference count. The rule applies to the journal’s own papers and to the world baseline.
• The narrow rule: the Leiden rule, applied only where the paper’s publisher has matched references for at least 50% of its papers that year. A publisher that deposits no references then does not lose its papers.
• The abstract rule (R8+clin): papers with a supplement signal, plus the narrow rule’s publisher-coverage test applied only in Health Sciences papers and journals (defined under Terms used). Release 2025 introduced this rule; this release applies it. The methodology change log summarises the precision check behind it: 93.0% of a sample of the papers it leaves out are not research articles (95% interval 89.4–95.5%).
• R5, an earlier candidate: the coverage test in every field, for journals whose publisher OpenAlex records, plus the supplement signals.

We computed the same comparisons with SNIP for all of them, on the same journals.

The abstract rule’s overall gain is smaller than the Leiden rule’s because outside Health Sciences it leaves out only papers with a supplement signal; there it changes OJR’s agreement with SNIP very little (+0.002 in the Social and Physical Sciences). In Health Sciences and Medicine all the rules give about the same gain.

The other agreement measures move the same way.

• Within subfields (2024, journals with every rule score): the median ρ with SNIP rises from 0.829 to 0.849 under the abstract rule (R5 0.853, Leiden 0.853, narrow 0.851). In Health Sciences it rises from 0.825 to 0.874 (R5 0.874, Leiden 0.882, narrow 0.876).
• Quartiles (2024), both recomputed on matched journals: overall, weighted κ rises from 0.642 to 0.661 under the abstract rule (R5 0.665, Leiden 0.665, narrow 0.666), and exact agreement from 61.2% to 62.7%. In Health Sciences, κ rises from 0.642 to 0.704 (R5 0.705, Leiden 0.700); exact agreement from 61.5% to 65.9%; within-one agreement from 94.7% to 97.3%.
• Norwegian Level 2 (2024): within-field AUC rises from 0.785 to 0.793 under the abstract rule (+0.008, 0.005–0.011), to 0.796 under R5 and to 0.798 under the Leiden or narrow rule. In Medicine it rises from 0.768 to 0.854 under the abstract rule.

Conclusion for the method decision

• Excluding zero-reference papers raises agreement with SNIP modestly overall and by about 0.06 in ρ in Health Sciences. Because Scopus already leaves most meeting abstracts out of its citable types, part of this rise is expected by construction. The abstract rule gives almost all of the Health Sciences gain and about two thirds of the overall gain.
• It also raises agreement with expert judgement, most of all in Medicine.
• No domain shows a clear loss. Under the abstract rule the smallest change is in the Social Sciences for 2025 (+0.001, −0.001–0.004; table above).
• The rules differ in what they protect: the narrow rule, R5 and the abstract rule keep the papers of publishers that deposit no references, and the abstract rule also leaves zero-reference papers outside Health Sciences alone unless a supplement signal applies. On this evidence that costs little agreement in Health Sciences and some overall (0.017 against 0.027 for the Leiden rule).
• Better agreement with SNIP is therefore a consistency check, not independent proof that the rule is right.

The choice of rule rests on the precision checks summarised in the methodology change log, which this analysis does not replace.

Data credits

• Computed from CWTS Journal Indicators © Centre for Science and Technology Studies (CWTS), Leiden University, based on Scopus (Elsevier). This article reports aggregate statistical comparisons and does not redistribute journal-level SNIP or IPP data.
• Norwegian Register for Scientific Journals, Series and Publishers © Directorate for Higher Education and Skills (HK-dir), licensed under CC BY 4.0^[8]; data reformatted into aggregate statistics.
• SJR data: SCImago. SJR — SCImago Journal & Country Rank. Retrieved October 2026, from https://www.scimagojr.com^[10]; obtained through the sjrdata R package mirror^[9]. This article reports aggregate statistical comparisons and does not redistribute journal-level SJR data.
• OJR is computed from OpenAlex (CC0).
• Scopus is a trademark of Elsevier. CWTS, SCImago and Elsevier have not reviewed or endorsed this study.

References

The DOIs were resolved against OpenAlex on 20 October 2026, and each work’s title, first author and year were checked. The figures cited from earlier studies are the ones our literature review reports. Datasets, documentation and software are labelled.

1. Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AFJ (2011). Towards a new crown indicator: an empirical analysis. Scientometrics 87(3):467–481. https://doi.org/10.1007/s11192-011-0354-5
2. Salvador-Oliván JA, Agustín-Lacruz C (2015). Correlación entre indicadores bibliométricos en revistas de Web of Science y Scopus. Revista General de Información y Documentación 25(2):341–359. https://doi.org/10.5209/rev_RGID.2015.v25.n2.51241
3. González-Pereira B, Guerrero-Bote VP, de Moya-Anegón F (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. J Informetrics 4(3):379–391. https://doi.org/10.1016/j.joi.2010.03.002
4. Cohen J (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213–220. https://doi.org/10.1037/h0026256
5. Hanley JA, McNeil BJ (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1):29–36. https://doi.org/10.1148/radiology.143.1.7063747
6. Moed HF (2010). Measuring contextual citation impact of scientific journals. J Informetrics 4(3):265–277. https://doi.org/10.1016/j.joi.2010.01.002
7. Waltman L, van Eck NJ, van Leeuwen TN, Visser MS (2013). Some modifications to the SNIP journal impact indicator. J Informetrics 7(2):272–285. https://doi.org/10.1016/j.joi.2012.11.011
8. Directorate for Higher Education and Skills (HK-dir) (n.d.). Norwegian Register for Scientific Journals, Series and Publishers. kanalregister.hkdir.no. Dataset, CC BY 4.0. https://kanalregister.hkdir.no
9. Kashnitsky I (2026). sjrdata: SCImago Journal & Country Rank data. R package, version 1.0.25. Software. https://github.com/ikashnitsky/sjrdata
10. SCImago. SJR — SCImago Journal & Country Rank. scimagojr.com. Database. Accessed October 20, 2026. https://www.scimagojr.com
11. CWTS (n.d.). CWTS Journal Indicators: Methodology. journalindicators.com. Documentation. Accessed October 20, 2026. https://www.journalindicators.com/methodology
12. Sivertsen G (2018). The Norwegian Model in Norway. J Data Inf Sci 3(4):3–19. https://doi.org/10.2478/jdis-2018-0017
13. Ahlgren P, Waltman L (2014). The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments. J Informetrics 8(4):985–996. https://doi.org/10.1016/j.joi.2014.09.010
14. CWTS (n.d.). Leiden Ranking Open Edition: Indicators. open.leidenranking.com. Documentation. https://open.leidenranking.com/information/indicators