0% READ
FULL TEXT
INTRODUCTION
The application of Bibliometrics to the field of Health Sciences enables one to analyze vast amounts of publications and their production patterns.[1] Success in the process of transferring research findings to the arena of medical practice depends largely on the methodological rigor of the studies and the speed in publishing results. To ensure the quality of output, a number of recommendations or guidelines have been developed, e.g. ICMJE, CONSORT, Clinical Trials Register (CTR), ARRIVE, PRISMA, STROBE, MOOSE and the EQUATOR network. The first three of these serve most often as the reference. Deserving mention is the interesting paper assessing CONSORT adherence in randomized clinical trials of disease-specific clinical decision support.[2]
The aim of the present paper is to classify journals belonging to a certain field in a set of thematic categories following a scientific criterion, then analyze the degree of adherence to the three mentioned guidelines (ICMJE, CONSORT and CTR). To this end, this paper proposes an original methodology consisting of estimating a multinomial logistic model with the category as a response variable and the guidelines as explicative factors. To fortify the model and make the response more precise, an additional factor was whether the journal was published under the Open Access system. The methodology was applied to a specific scientific category registered in the 2023 edition of the JCR, hence dealing with data from the year 2022, namely Dentistry, Oral Surgery and Medicine, and the journals were broken down into three major thematic blocks. The category selected in this study is held to be a global representation of Medicine yet limited to a specific area; in fact, its journals take in very diverse fields, from basic science to clinical and community studies, showing an evolution similar to that of Medicine.[3] At the same time, its self-citation rate indicates a healthy scientific publishing environment.[4]
A previous work developed the initial stage of an estimation system of by means of ordinal regression to predict the tercile that a journal would occupy according to its Journal Impact Factor (JIF), taking into account diverse sets of factors and covariables;[5] in a second stage, the JIF itself was estimated through a multiple linear model.[6, 7] At that time, however, the authors did not appraise how the the degree of adherence to publication guidelines might influence the variable of response.
Within the realm of logistic regression models in Scientometrics, a recent study applied logistic regression (among other techniques) to develop an automated contextual classifier for scholarly papers by utilizing established algorithms and understanding the information retention of different parts of a scholarly article, such as the Abstract, Article Title, and Keywords.[8]
Also deserving mention are recent contributions for modeling the life cycles of several trending topics based on their frequency over time in published abstracts using the growth rate of topics as an explanatory variable to estimate the probability of being lowly cited versus modestly or highly cited in research papers.[9, 10] Previously, a paper by Boyack et al., developed a multinomial logistic regression model to classify individual articles by research level.[11]
Meanwhile, the number of journals that publish-fully or partially-under the system of Open Access has vastly grown. This permits wider diffusion of research findings, especially in developing countries, and their influence upon quality indicators has been amply studied.[12, 13]
Within the field of Dentistry, an earlier study about the degree of adherence to author guidelines in dental journals concluded that journals with higher impact factors have more rules guiding publication policy.[14]
The paper is structured as follows: First, the three most important quality guidelines are described, and then the statistical methodology applied for analysis is outlined. After expounding the results obtained, they are discussed, and the main conclusions of the study are highlighted. The degree of adherence to the guidelines of reference of CONSORT and CTR are the significant factors to be taken into account.
AN OVERVIEW OF THE MAIN RECOMMENDATIONS AND GUIDELINES
As mentioned above, this study explored the following three sets of guidelines to improve quality of publications in medical journals:
International Committee of Medical Journal Editors (ICMJE)
The ICMJE put out a series of relevant recommendations for the first time in 1978 as a means of standardizing the format of scientific papers before their publication in Health Science journals. They have since been followed by many biomedical journals. The norms proposed by the ICMJE, approved by over 2600 biomedical journals worldwide, are underlined in their author instructions.[15] Accordingly, the most recent version of www.icmje.org, should be cited as a source, because the ICMJE updates the reccomendations periodically. The most recent update is that of 2021.
A further aspect of concern is any potential conflict of interest, to be addressed as part of the authors´ responsibilities, given that papers must demonstrate the greatest transparency possible, the reliability/credibility of articles published being of foremost importance for the readership. The ICMJE also informs about how peer-review should be conducted.
CONsolidated Standards Of Reporting Trials (CONSORT)
CONSORT considerations, established in 1996, lay out a series of 25 points considered essential to carry out and publish a sound random clinical trial. The aim is to help authors improve the quality and precision of the publications deriving from this particular type of biomedical study. Since the CONSORT norms have been recommended and/or required by hundreds of medical journals, including those of high impact, and by publishing groups such as the aforementioned ICMJE.[16]
A study by Plint et al., determined the impact of CONSORT recommendations in randomized clinical trials published in medical journals, concluding that they helped enhance the quality of such reports, thus confirming their utility.[17] CONSORT is updated from time to time, most recently in 2010 (www.consort-statement.org). The CONSORT requirements apply to all sections within the design and publication of clinical trials, including: Title, Abstract, Methods, etc.
Clinical Trials Register (CTR)
The importance of clinical trials in the pyramid of evidence makes their register a scientific, ethical and moral imperative. As stated in 2018 by the World Health Organization,[18] and by the Declaration of Helsinki “Each clinical trial should be registered in a database of public access before recruiting the first subject”. A growing number of institutions offer researchers (or other interested individuals) free-access web pages where clinical trials are registered.
METHODOLOGY
The multinomial regression model
Multinomial regression is an extension of binary logistic regression with one or more independent variables, in cases where the response or dependent variable presents more than two categories, as in this study, in which the journals were classified into three categories based on their content. So, let Y be a polytomic response variable that can take on more than two categories Y1, Y2,…, Ys, and be modeled in terms of a set of explicative variables X1, X2,…, Xk. The multiple response model adapts the probability of each one of the response categories in terms of the observations of the explicative variables in a generic way as πs(x)=Prob(Y=Ys/X=x), s=1,2,…,S, for each vector x of observed values of the explicative variables,[17] verifying that Σπs(x)=1[19]
In this study we are interested in modeling the category to which a variable pertains (C1, C2,…, Cs) in view of the factors: adhesion to ICMJE, to CONSORT, to CTR and Open Access. The multiple response model estimates the probability of each category for each value observed in the explicative variables.
For each vector of values observed among the explicative variables x=(x0, x1, …, xk)T with x0=1 and given the vector of parameters β=(β0s, β1s, …, βks)T associated with the category , denoting:
the model can be expressed in terms of the probabilities of response:
The parameters of the multiple response model are estimated by means of the maximum likelihood method, based on the multinomial distribution. The properties of the estimators of the S-1 parameter vectors are inherited from the maximal reliability estimation: asymptotic un-biased, asymptotic Gaussian distribution, and minimum variance. Therefore, all methods of inference regarding the parameters of the model are either based on the asymptotic properties of the parameters or else on the distribution of the Wilks statistic of likelihood ratio.
By applying the likelihood ratio test, the influence of each factor on the complete model using all of them was analysed.[20] Thus, taking as the null hypothesis that the parameter of a certain effect is 0, the chi-square statistic is the difference in -2 log (likelihoods) between the final and the reduced model (i.e. formed by omitting the effect from the final model).
The level of performance of the model was appraised by the Correct Classification Rate (CCR), which represents the percentage of journals that the model classified correctly, and by the Nagelkerke coefficient, a logarithmic scoring rule based on the log-likelihood that can serve as a measure of overall model performance.[21]
Computational treatment was carried out with R (The R Project for Statistical Computing).
Application to Dentistry journals
The methodology described above was applied to the Journal Citation Report field Dentistry, Oral Surgery and Medicine, including 92 journals in its 2021 edition. Although it may appear somewhat subjective, the classification by subject was strictly carried out according to the following coding:
A: Of a general scope, including Pediatric Dentistry and Gerodontology.
B: Periodontics, Oral Surgery and Implantology, and Oral Medicine.
C: Orthodontics, Restorative, Endodontics, Prosthodontology, Public Health and others.
The journals were ordered according to their Journal Citation Indicator (JCI), to be divided into terciles; then a stratified random sampling provided 10 from each group, thus giving a total of 30.
For the sampled journals, the corresponding web pages were accessed to consult the instructions to author or author guidelines. The degree of adherence of a journal to each of the three guidelines was evaluated by means of the following nominal scale:
0: makes no mention.
1: following them is recommended.
2: it is obligatory to adapt to the guidelines.
Of the sample of 30 selected journals, all but two (i.e. 93.33% of the total) contained instructions to authors, so it was necessary in those two cases to contact the Editor-in-Chief by email to request them and inquire about the journal's level of adherence to the ICMJE, CONSORT, and CTR. A response was received in both cases, and the information provided was initially considered accurate. As a step previous to the analytical treatment, descriptive tables indicating frequencies were elaborated.
An additional factor made reference to whether the journal published articles under Open Access: No (0) / Yes (1), information extracted from the Scientific Journal Rank, 2023 edition. Finally, a multinomial logistic model for the classification variable was estimated, taking into account both the Nagelkerke coefficient and the CCR evaluated.
RESULTS
Data description
Table 1 lists the journal sample obtained from the JCR for Dentistry, Oral Medicine and Surgery in 2020 along with the JCI tercile that the journal occupies, the category of classification, the level of demand reflected by the recommendations and guidelines included in this study, and the character of Open Access (or not).[22] The classification of journals by thematic field gives a fairly even distribution over the three groups defined: 30.0% in category A, 36.7% in B, and 33.3% for C. Yet only 10% of them were Open Access journals. The graph showing the precision of prediction of the category in each of the three cases can be seen in Figure 1.
| Journal | T-JCI | ICMJE | CONSORT | CTR | OA | Categ | Estim | Prob |
|---|---|---|---|---|---|---|---|---|
| J. Clin. Periodontol. | 1 | 2 | 2 | 2 | 0 | B | B | 0.774 |
| J. Dental Res. | 1 | 2 | 2 | 2 | 0 | A | B | 0.774 |
| J. Periodontol. | 1 | 2 | 2 | 2 | 0 | B | B | 0.774 |
| J. Endodontics | 1 | 2 | 2 | 0 | 0 | C | C | 0.5 |
| Clin. Oral Implants Res. | 1 | 2 | 2 | 2 | 0 | B | B | 0.774 |
| Oral Oncol. | 1 | 2 | 2 | 2 | 0 | B | B | 0.774 |
| J. Am. Dent. Assoc. | 1 | 2 | 2 | 2 | 0 | A | B | 0.774 |
| Int. J. Paediatric Dent. | 1 | 2 | 1 | 1 | 0 | A | A | 0.645 |
| Eur. J. Oral Sci. | 1 | 2 | 1 | 1 | 0 | A | A | 0.645 |
| Oper. Dent. | 1 | 2 | 0 | 0 | 0 | C | C | 0.545 |
| J. Adhes. Dent. | 2 | 1 | 0 | 0 | 0 | C | C | 0.545 |
| Dent. Traumatol. | 2 | 2 | 1 | 2 | 0 | C | C | 0.5 |
| Int. J. Oral Maxillofacial Surg. | 2 | 2 | 2 | 1 | 0 | B | A | 0.71 |
| Int. Dental J. | 2 | 2 | 1 | 1 | 1 | A | A | 0.645 |
| Acta Odontol. Scand. | 2 | 1 | 0 | 0 | 0 | A | C | 0.545 |
| J. Cranio-Maxillofacial Surg. | 2 | 2 | 1 | 2 | 0 | B | C | 0.5 |
| J. Adv. Prosthodont. | 2 | 2 | 0 | 0 | 1 | C | C | 0.545 |
| Int. J. Dent. Hyg. | 2 | 2 | 1 | 1 | 0 | C | A | 0.645 |
| Med. Oral Patol. Oral Cir. Bucal | 2 | 0 | 0 | 1 | 1 | B | B | 1 |
| J. Oral Maxillofacial Surg. | 2 | 1 | 2 | 2 | 0 | B | B | 0.774 |
| Int. J. Periodont. Restor. Dent. | 3 | 0 | 0 | 0 | 0 | B | C | 0.545 |
| Br. Dent. J. | 3 | 0 | 2 | 1 | 0 | A | A | 0.71 |
| Aust Endod J. | 3 | 1 | 1 | 0 | 0 | C | C | 1 |
| Int. J. Prosthodont. | 3 | 1 | 0 | 0 | 0 | C | C | 0.545 |
| Quintessence Int. | 3 | 0 | 2 | 0 | 0 | A | A | 0.5 |
| J. Orofac. Orthop. | 3 | 0 | 0 | 0 | 0 | C | C | 0.545 |
| J. Oral Implantol. | 3 | 0 | 0 | 0 | 0 | B | C | 0.545 |
| J. Stomat. Oral Maxillofac. Surg. | 3 | 0 | 0 | 0 | 0 | B | C | 0.545 |
| J. Clin. Pediatr. Dent. | 3 | 0 | 0 | 0 | 0 | A | C | 0.545 |
| Semin. Orthod. | 3 | 0 | 0 | 0 | 0 | C | C | 0.545 |
Furthermore, to assess the association between the journal's tertile and the degree of adherence to the three guidelines considered, an overall score was introduced. This sum consists of the individual scores and defines a scale of: Low (when the sum is 0, 1, or 2), Medium (when it is 3 or 4), and High (when it is 5 or 6). The contingency table is shown in Table 2, which provides a chi-square value: χ2 = 13.6 (p=0.009), indicating that the association between the two characteristics is significant. A graphic representation is offered in Figure 2.
| Score | Low | Medium | High |
|---|---|---|---|
| 1st tercile | 1 | 3 | 6 |
| 2nd tercile | 4 | 2 | 4 |
| 3rd tercile | 9 | 1 | 0 |
Table 3, in turn, gives the percentage of journals that follow each degree of adherence to the three sets of guidelines. As seen, over half the journals sampled required that the ICMJE norms be followed. In contrast, the percentage of journals that do not mention the CTR, or merely recommend it, amounts to 70%.
| Degree of adherence to quality guidelines | ICMJE | CONSORT | CTR |
|---|---|---|---|
| Not mentioned (0) | 30.0% | 40.0% | 46.7% |
| Recommended (1) | 16.7% | 23.3% | 23.3% |
| Required (2) | 53.3% | 36.7% | 30.0% |
Results of the multinomial regression
To appraise the influence of factors upon the multinomial model with the response variable journal category, a likelihood ratio test was performed. It gave the significant values shown in Table 4, considering, in the first place, all the factors. In a second run, it generated only those that were found to have a significant influence.
| Factor | First step (p-values) | Second step (p-values) |
|---|---|---|
| ICMJE | 0.137 | |
| CONSORT | 0.059 | 0.033 |
| CTR | 0.003 | 0.012 |
| Open Access | 0.607 |
Therefore, we conclude that the influential factors in the variable of response were: the adherence to the recommendations made by the CONSORT and the Clinical Trials Register. The Nagelkerke coefficient associated with the final model is RN2=0.623 and the correct classification rate is 66.7%, as indicated in Table 1, where capital letters in blue indicate agreement between the actual and estimated categories, while red represents discrepancies.
DISCUSSION
Although the main interest of this paper may appear limited to the objective of developing a statistical model that allows one to classify journals in a series of categories in terms of their adherence to three sets of author guidelines, the methodology applied serves to test the performance of journals according to other categories. Broader categories may be considered, or even some other specific classification system, such as the JCR, to draw up a set of predetermined factors and covariables.
A first result to take into account is the proven existence of a significant association between the tertile occupied by a journal and the overall degree of adherence to the guidelines. On the other hand, the percentages of journals that do no mention guidelines or merely recommend them is similar among the three groups (p=0.706). The degree of adherence to the CTR guidelines is indeed significant (p=0.042) for the journals of this group with regard to those of the other two groups, and the high percentage of journals in group C that do not even recommend them is noteworthy. This result may be attributed to the fact that the ICMJE deems it necessary for clinical trials to be registered, at their conception, in public databases accessible for authors, researchers, and regulating agencies.[23] Ultimately, because the credibility of scientific articles published relies on the rigor and adequacy of the methodology applied, the results and additional information generated by scientific research should be made freely available to the whole international community for reference and critical appraisal.
It is important to add that, despite the relevance of the ICMJE recommendations within the clinical field, this factor presents a significant association with the CONSORT considerations (p=0.018) and, above all, with CTR (p=0.015), for which reason it is excluded from the multinomial model.
Clinical research analyzing the extent to which CTR related to temporomandibular disorders adhered to the CONSORT statement, reported a mean adherence of 71%.[24] A study previously cited[15] appraised the adherence of journals in the field of Oncology and Hematology to the guidelines of ICMJE, CONSORT, MOOSE, PRISMA, STARD and STROBE. Mentioned the most in this case were the ICMJE declaration, the CONSORT checklist and the CTR (respectively 67.5 %, 33.3% and 42.9%).
Meanwhile, analysis of the adherence of random controlled trials on diabetic retinopathy gave a range of adherence from 24% to 68%.[25] Similarly, the study by Plint et al., determined whether adopting the CONSORT verification list was associated with an improvement in the quality of the reports on randomized controlled trials.[17]
Evaluating the integrity of published trials, that is, those fulfilling the CONSORT verification list, is important for directing publication policies and minimizing the risk of a selection/publication bias.[26]
CONCLUSION
We may conclude the following points:
- For the three categories or thematic blocks defined in this study, the degree of adherence to the guidelines of reference of CONSORT and the Clinical Trials Register would contribute to properly classifying a journal in a thematic category, unlike what happens with ICJME.
- The dental subjects in group C (Orthodontics, Restorative, Endodontics, Prosthodontology, Public Health and others) share similar editorial policies.
- The Open Access publication system is not linked to any particular category within the field of Dentistry.
- Journal editors should ensure that their Instructions to Authors contain adequate information on relevant recommendations and guidelines.