Appraise

5.3 Other sources of bias

Reporting issues

As part of assessing the quality of a paper, the reader needs to evaluate how the author has reported the methods and results of their study. Careful appraisal of a paper may leave you uncertain about whether basic concepts of the study design were duly considered when planning and conducting the study, but have simply been poorly reported.

If you look through the literature, you will find articles that are biased: by poor reporting of crucial information e.g. age and medical history of the enrolled animals; by inappropriate definitions or diagnoses of diseases; or by a lack of (or inappropriate) control groups (Dean, 2013).

Poor reporting

Poor reporting reduces the transparency of research and limits the reader’s ability to critically appraise information because information that has not been included cannot possibly be appraised! The descriptions included in the paper should allow the reader to repeat the study in order to attempt to obtain an independent result. Examples of important deficits that may be found in veterinary literature are: missing information of the type of animals used in the study and how they were allocated; unclear description of diagnostic methods; and inappropriate documentation of treatments and outcome measurements.

Ultimately, if certain information is not given in a paper, you should regard this as not having been considered in the study design or study implementation. It is better to be safe than to be sorry when appraising literature that could inform important decisions you make about your patients!

Reporting guidelines

Reporting guidelines (for example STROBE-VET ) exist to guide authors and publishers of journals to ensure that papers are written with sufficient transparency and clarity. However, not all veterinary journals refer to reporting guidelines (Grindlay et al., 2014 reported the figure to be as low as 35%). It is important to note that reporting guidelines are different from critical appraisal toolkits, which assist readers to determine whether the evidence presented within a published paper is of good quality.

Peer review

Peer review has been the quality control process for scientific publishing for many years, purportedly ensuring that information is checked and verified by subject experts before it is formally published. This saves the reader time; the onus is not on the reader to conduct the only fundamental analysis of the quality, accuracy and validity of the content. Peer-reviewed publications from the scientific and veterinary communities are key sources of information for EBVM practitioners.

However, some limitations and possible biases of peer review have been identified (Benos et al., 2007). For example, it has been demonstrated that gender and affiliation of the authors had an impact on the review outcomes. It is important to remember that peer review is not perfect, and published peer-reviewed studies vary in quality. However, studies show that manuscripts improve considerably after the peer-review process (Goodman et al., 1994; Benos et al.,2007).

Publication bias

Traditionally, peer-reviewed scientific journals and the bibliographic databases that index them have been considered the best sources of evidence, but research into publication bias (Glanville et al., 2015) suggests that there is a need to go beyond these sources, because a significant proportion of research will not be published in peer-reviewed journals.

Publication bias occurs when researchers, or journal editors, decide to publish studies with ‘positive’ or statistically significant results (for example, showing that a treatment has a beneficial effect) but do not publish those with no ‘significant’ results (for example, when a treatment had no beneficial effect), despite it being a well-designed study. If this happens, analysis of the published results will not provide an accurate representation of current evidence.

This publication bias is perhaps particularly relevant in the field of clinical veterinary medicine, where practitioners may not be publishing their work as peer-reviewed articles, and much of the scientific data may be hidden in the so-called ‘grey’ literature (e.g. conference papers), or in practice records and case reports. For more information about finding this ‘grey’ literature, see Acquire: What sources of evidence are there?

Sponsorship bias

Finally, check who funded the study. If it is, for example, a pharmaceutical company, the study may suffer from sponsorship bias which may lead to poor reporting e.g. not all results may be presented (Wareham et al., 2017). In general, not every sponsored project provides biased data, but you should carefully consider the quality criteria if you think that the study sponsor may have a vested interest in what and how the results are reported.

Predatory journals

As mentioned in Acquire: Internet search tools, disreputable online publishers, sometimes referred to as 'predatory', have emerged in recent years. They exploit the open access model of publishing where the author pays a fee (an Article Publishing Charge or 'APC'). The disreputable publisher takes the money but fails to follow through with the peer-review and editing process that is the standard expected from a reputable scientific journal. This has led to a proliferation of freely available poor-quality research and although these would not be listed by databases, such as MEDLINE, they will be found by Google searches.