0% READ
FULL TEXT
Bibliometrics, which began as a tool for evaluating journals through citation analysis (Garfield, 1972), has over the past two decades crossed disciplinary and institutional boundaries in ways its early practitioners could scarcely have imaged. Today, bibliometric tools and indicators sit at the heart of Science, Technology, and Innovation (STI) policy, informing decisions on research funding, national competitiveness assessments, university rankings, and the evaluation of individual researchers (Aksnes et al., 2019; World Intellectual Property Organization, 2025). This expansion is, in many respects, a testament to the field’s maturity and relevance. But it also brings with it a risk that Goodhart’s Law captures precisely: when a measure becomes a target, it ceases to be a good measure. Bibliometric, or more precisely bibliometric indicators have in many settings, become ends in themselves, pursued at the expense of the underlying scientific contributions they were meant to reflect. The very features that have made bibliometrics attractive to policymakers, namely its apparent objectivity, its quantitative precision, and the relative ease with which its tools can be applied, are the same features that make it vulnerable to misuse. Citation counts, Journal Impact Factor (JF), h-index, and are now routinely used in contexts and at scales their originators never intended (Hicks, Wouters, Waltman, de Rijcke, and Rafols, 2015). Bibliometric software platforms such as VOSviewer, Biblioshiny have further lowered the barrier, allowing users to construct sophisticated-looking analyses, network maps, and visualizations with minimal methodological training. The concern is not that these tools exist or that they are widely used. What is problematic is the growing tendency to apply these tools without adequate understanding of their assumptions, limitations, and appropriate scope. Empirical findings from two global surveys conducted by Lancho Barrantes et al. (2026) draws attention to this; showing that 76% in 2020 and 70% in 2022 of primarily practitioners reported having no formal training in bibliometrics education. Across academia and policy circles, are witnessing a growing desire to produce objective judgments on research quality, institutional performance, and national scientific output, and bibliometrics is being reached for as the instrument of choice. In many of these instances, the tools are being used in ways that are, at best, methodologically questionable and, at worst, actively misleading. Issues such as differences across research fields in how citations accumulate are ignored. A single number is used as a stand-in for a complex, multidimensional achievement. Composite indicators are built without transparency about what went into them. The result is not objectivity. It is the appearance of objectivity, which is considerably more dangerous.
The consequences of misusing bibliometric tools are not abstract. When flawed indicators drive funding decisions, resources go to the wrong places. When researchers are judged on measures that do not reflect the nature or value of their work, careers are distorted and incentives are corrupted. When national STI policies are built on poorly constructed evidence of this kind, entire research systems can be pushed in counterproductive directions (Wilsdon et al., 2015). As the CoARA Steering Board has noted, overreliance on even well-intentioned bibliometric assessments can have damaging trickle-down effects (Balboa et al., 2024). Perhaps most troublingly, misuse erodes trust, both in the indicators themselves and in the scientific enterprise they are meant to serve. When bibliometrics is seen to produce unfair or inaccurate assessments, the response is often not to improve the methodology but to abandon evidence-based evaluation altogether. That would be a serious loss for research governance. Yet fairness concerns are not always the result of misuse. Some are structural. The Helsinki Initiative on Multilingualism in Scholarly Communication (2019) (Federation of Finnish Learned Societies and The Committee for Public Information, 2019) drew pointed attention to one such structural issue: that research evaluation and funding systems fail to adequately reward multilingual scholarly communication, and that language diversity must be actively promoted in how research is assessed and funded. Compounding this, the dominant bibliometric databases give systematic priority to English-language journals, meaning that research published in other languages remains structurally undercounted regardless of its quality.
The effect of artificial intelligence on the integrity of the scholarly record is a challenge the bibliometric community has yet to address with the seriousness it deserves. Large language models are increasingly used to draft manuscripts and generate reference lists, and when these outputs contain fabricated citations, the databases on which bibliometrics depends are directly contaminated (Walters and Wilder, 2023). AI-assisted publishing has also accelerated the growth of low-quality journals that artificially inflate the bibliometric footprint of those who use them strategically. This is not a peripheral concern; it goes to the foundation of what the field is measuring. Yet AI can also be used in a positive way. Machine learning tools are already being deployed to detect anomalous citation patterns, identify paper mills, and flag potential data manipulation, capabilities that, properly developed, could strengthen rather than undermine bibliometric integrity (Notman, 2025). The field’s task is to engage critically with these tools rather than either ignore or uncritically embrace them.
Responsible Bibliometrics
It is in this context that the concept of responsible bibliometrics becomes essential, not merely as a reform slogan, but as a practical standard for how this field conducts and communicates its work. The idea builds on what the Metric Tide report (Wilsdon et al., 2015) called responsible metrics, five core principles: robustness, humility, transparency, diversity, and reflection on unintended consequences. These principles insist that measurement should be grounded in the best available data, that numbers should support rather than replace expert judgment, that methods should be open to scrutiny, that a range of indicators should be used to reflect the diversity of research, and that those who design and use metrics should remain alert to their wider effects. Rushforth and Hammarfelt (2023) have traced how this responsible metrics agenda has developed as an organized effort at professional reform. They identify three broad positions within it: those who are broadly skeptical of metrics, those who argue for greater professional expertise in their use, and those who call for ongoing, honest reflection by the field on its own practices. While each position makes a different argument, all three share a common underlying concern: that bibliometrics be used with sensitivity to disciplinary context, awareness of its limits, and honesty about what it cannot tell us.
A useful way to understand this comes from recent scholarship that frames bibliometric indicators as the “blood tests” of research systems (Cicero, 2026), diagnostic tools that can reveal important things about the health of a research system, but only when interpreted by trained professionals who understand what the numbers do and do not mean. A physician does not diagnose from a single test result in isolation. A research evaluator should not judge a career, an institution, or a national research system from a citation count alone. Nor should evaluation rest on peer review alone: that process, for all its acknowledged strengths, is subject to bias, inconsistency, and opacity, and is costly and time-consuming to conduct at scale. As Belter (2015) argues, the strengths of bibliometric indicators correspond directly to the weaknesses of peer review, and the two methods, used together, offer a more balanced basis for evaluation than either can provide alone. The demand, then, is not for the abandonment of quantitative tools, but for their disciplined and contextualised use alongside expert judgment. The challenge is that this combination remains more aspiration than practice in most institutional settings. Despite widespread formal endorsement of its principles, the day-to-day reality of research evaluation continues to be shaped by crude indicators applied without contextual sensitivity. The gap between what the field knows and what institutions do is itself a subject that demands urgent and sustained scholarly attention. Kar (2025) argues that institutions and funding bodies should adopt a diverse set of indicators, rather than relying on a one-size-fits-all model. It should include a balanced mix of qualitative peer review, societal impact narratives, policy influence, interdisciplinary engagement, methodological rigor, and adherence to open science practices to holistically evaluate the multifaceted nature research performance. Moreover, integrating principles of Scientific Social Responsibility further strengthens this approach by recognizing researchers’ contributions to community engagement, ethical conduct, knowledge equity, and the broader public value of science, thereby enabling a more comprehensive and context-sensitive assessment framework. Expanding this further, the Altmetrics Manifesto proposed that the dissemination and societal reach of research, measurable through mentions in policy documents, media coverage, and engagement on digital platforms, should complement citation-based indicators in any comprehensive assessment framework (Priem et al., 2010).
Reforming Research Assessment: Initiatives and Debates
The field has not lacked for reform initiatives, though their origins differ significantly. The San Francisco Declaration on Research Assessment (DORA) (DORA, 2013), first published in 2013 and now endorsed by over 25,000 individuals and organizations worldwide, called explicitly for an end to the use of JIF as a shorthand for individual research quality. It went further, urging that research assessment consider the value of all research outputs beyond journal articles, including datasets and software, and that a broad range of impact measures be used, including qualitative indicators such as influence on policy and practice. The Coalition for Advancing Research Assessment (CoARA) (“Coalition for advancing research assessment”, 2022), established in 2022, has brought together over 700 organizations committed to reforming research evaluation in Europe and beyond. Both initiatives were driven primarily by research institutions, funders, and publishers rather than by the evaluative scientometrics community itself.
The bibliometrics community also has not been passive in this debate. The Leiden Manifesto (Hicks et al., 2015), published in Nature in 2015 by a group of leading bibliometricians, set out ten principles for responsible bibliometric practice that remain as relevant today as when they were written. However, there has been concerns of a growing trend of rejecting bibliometric indicators in research evaluation. Torres-Salinas, Arroyo-Machado, and Robinson-Garcia (2023) argue that it stems from misinterpretations of the DORA declaration and encourages evaluation by qualitative measures only, neglecting the supportive role of bibliometrics. Abramo (2024) expresses concern that CoARA bypasses the established evaluative scientometrics community, lacks contextualization, and proposes a standardized approach where the diversity of research contexts demands the opposite. That the International Society for Scientometrics and Informetrics had no involvement in CoARA’s formation is, for Abramo, a sign of a deeper disconnect between science policy and the science of research evaluation. More recently, leading scientometricians including Daraio, Glänzel, and Gorraiz have called for a structured framework to determine whether any given indicator is fit for purpose in a specific evaluation context, arguing that the misuse of indicators is the problem, not the indicators themselves (Daraio et al., 2025). The field’s leading journal, Scientometrics, is currently publishing a dedicated special issue on the Framework for the Responsible use of Assessments and Metrics in Evaluation, bringing together research on how bibliometric indicators should be selected, applied, and interpreted responsibly across different evaluation contexts. This follows a special track on the same theme at ISSI 2025, the field’s premier international conference. What remains clear is that the field is no longer debating whether reform is needed, but about how far it should go and who should lead it.
In a recent blog post (Balboa et al., 2024) and the CoARA Steering Board has underscored that important facet of CoARA’s implementation is the facilitation of mutual learning and exchange on evaluation practices where qualitative and quantitative approaches are meaningfully combined. This provides a new pathway forward toward meaningful exchange between research assessment reform community and scientometric community toward developing common tools sets. An important work in this direction is the document prepared by the nor-CAM (Universities Norway, 2021) a working group appointed by Universities Norway, mandated to recommend guiding principles for the assessment and evaluation of researchers in light of the transition to Open Science has proposed guiding principles for new research assessment framework. Among others, they have presented how at different levels (macro, meso and micro levels), peer review and bibliometrics can enrich each other in the evaluation process. Another novel initiative is “narrative bibliometrics” proposed by Torres-Salinas, Orduña-Malea, Delgado-Vázquez, Gorraiz, and Arroyo-Machado (2024), defined as the use of bibliometric indicators to generate narratives and stories that support the defense and exposition of a scientific curriculum and/or its individual contributions within the framework of a scientific evaluation process. It is interesting that these intellectual discussions are taking shape toward developing a new generation of assessment frameworks that brings more trust in bibliometric indicators.
And yet, in most universities and funding bodies, the JIF, and h-index still dominates promotion panels. JIF in particular remains a hiring shorthand and quality judgment of researcher’s performance. Reform documents are endorsed and then set aside. This is not a failure of ideas. It is a failure of institutional will, and of incentive structures that reward compliance with existing metric conventions over honest critique of them. The consequences of this institutional inertia extend into scientific conduct itself. Pressure to publish, arising directly from overreliance on publication-based indicators for career assessment, is among the primary drivers of questionable research practices, including the pursuit of “me-too science” and a strong disincentive to pursue risky but potentially groundbreaking research (Bhattacharya, 2023). Reform of assessment systems and reform of research integrity are not separate conversations. They share the same root. The problem runs deeper than metric misuse alone. Bhattacharya (2020) has argued that divergent and selectively framed models of science and innovation policy are producing weak governance frameworks, ones that fail to account for the uncertainties, value disputes, and contested knowledge that characterize contemporary science. Applying bibliometric indicators designed for a narrowly framed model of science to a research landscape that is interdisciplinary, internationally distributed, and increasingly entangled with societal risk is precisely this kind of governance failure. Reform of evaluation frameworks and reform of the science policy paradigm are not independent tasks.
For policymakers and research managers, responsible bibliometrics has direct practical implications. Those commissioning evaluations should ask, as a minimum, whether the indicators used are appropriate to the field being assessed, whether more than one source of evidence has been brought to bear, and whether the methodology is transparent and independently reviewable. Funding bodies should adopt a diverse set of indicators rather than a single ranking or score. It is time for the bibliometrics community to take collective ownership of how its tools are used beyond its own borders, developing shared guidelines that are both field-sensitive and actionable, and treating the professionalization of research evaluation as a priority rather than an afterthought. Researchers too have a responsibility here. Gaming metrics, engaging in strategic self-citation, or participating in citation exchange arrangements all corrode the system. As Biagioli and Lippman (2020) argue “growing reliance on institutional metrics of evaluation does not just provide incentives for these kinds of manipulations, but also creates their conditions of possibility.” There is also a less visible problems that researchers face but remain silent, for example some reviewers use the peer review process to pressure authors into citing their own work, a practice that artificially inflates citation counts and corrupts the very indicators the field depends upon. Responsible bibliometrics must address researcher and reviewer behavior, not only the behavior of evaluators. The bibliometrics community did not create the demand for simple answers to hard questions about research quality. But it has, at times, been insufficiently clear about the limits of what its tools can responsibly deliver. That silence has created space for misuse. Filling that space with rigorous, accessible, and authoritative guidance is perhaps the most important contribution the field can make at this moment.
Bibliometrics stands at a genuine crossroads. Its influence has never been greater; the risks that come with that influence have never been more visible. The field has the intellectual tools, the accumulated empirical record, and the institutional standing to lead a more disciplined and responsible era of research evaluation. What it needs now is the collective will to do so. Ziman (2000) observed that “trust” is the foundation of scientific knowledge, and that the relations by which we hold knowledge have a moral character. The indicators through which bibliometrics represents scientific knowledge share that moral character. Using them well, and acknowledging honestly when they are being used badly, is not a technical matter alone. It is an ethical one. Science has long relied on references as markers of credibility, rooted in the norms described by Robert K. Merton and reinforced through citation systems developed by Eugene Garfield and subsequently sharpened, over decades, by a growing community of scholars who deepened its tools, techniques, and conceptual foundations. With the irresponsible application of bibliometric indicators, and the rise of AI, this foundation is being unsettled. This makes it imperative to rethink research evaluation, placing greater emphasis on verifiability and traceability rather than surface indicators of credibility.
It is in this spirit that the Journal of Scientometric Research reaffirms its commitment to being a space where these difficult conversations are held rigorously and openly, where indicators are debated, and evidence is scrutinized. Concretely, this means the journal will actively encourage submissions that address responsible bibliometrics from multiple disciplinary angles, including not only scientometrics but also science policy, innovation studies, research ethics, and information science. It will also, in the near term, convene a dedicated special issue on responsible bibliometrics, bringing together researchers, evaluators, and policymakers to examine what the principles mean in practice, how they can be operationalized across different institutional contexts, and where the field’s most urgent gaps lie. This is not a conversation the bibliometrics community can afford to leave to others. The alternative, continuing to allow questionable practices to proceed under the authority of bibliometric tools, is one the field can no longer remain silent about.