On the More Insidious Manifestations of Bias in Scientific Reporting

Beware the Adverbs (and Sometimes) Adjectives: Back to Basics 

The variable manifestations of classic scientific methodologic bias in the reporting of results have been previously summarized [3]. Notwithstanding this prior review, certain identifiable and some less obvious manners of reporting should alert readers and reviewers to an insidious bias that has escaped attention. The use of adverbs should have a restricted place in medical discussions; objective results deserve discussions that provide for a rationale argument, without editorial emphasis. In reviewing the literature regarding the potential role of CT colonography (CTC), the authors of one paper made the following statement [4]:

The natural history of small colorectal polyps has become a central issue of critical importance. One reason is that CTC is undoubtedly [emphasis added] a highly efficacious and cost-effective approach to population screening when only large polyps (≥10 mm) necessitate referral for polypectomy.

Such comments may be justified and will find a receptive audience among radiologists. The use of the word undoubtedly, however, is as unnecessary as it is incorrect. Policymakers in both government and private industry, who continue to decline to recommend reimbursement for CTC, have reviewed the same literature in context of both larger and smaller polyps (and the natural history associated therewith) and, by reaching a different conclusion, demonstrate “doubts” about the authors' position. The concern here is not whether one group or another is correct (witness the recent controversy of the US Preventive Services Task Force recommendations on screening mammography compared with those of the American Cancer Society to recognize how the same data lead to different conclusions) but rather the use of argumentative adverbs such as undoubtedly in overstating one perspective.

In an invited editorial regarding a study that my colleagues and I published last year regarding the disclosure of harmful errors by physicians to patients, the invited editorial commentator, in a thoughtful analysis, used the word unequivocal in relating how the law views the advisability of such disclosures [5, 6]. Beyond statutory law, in which provisions are expressly stated, few issues in law are “unequivocal,” and this is not one of them.

Consider a review of the regulatory obstacles involved in approving the use of contrast-enhanced ultrasound agents in the United States, when these agents have been approved in Europe and Asia [7]. It is not difficult to identify with the learned authors' perspective in pointing out that political aspects of approval may complicate an objective analysis. Nonetheless, when the text of the discussion enlists the word compelling to describe the efficacy of contrast-enhanced ultrasound, there is an inherent contradiction. The evidence, as “persuasive” as it may be, is, by definition, not sufficiently “compelling” to translate into approval by the US Food and Drug Administration.

Many journals, including Radiology (of which I am currently an associate editor), reject words such as significant in an adjectival manner and permit it only when describing statistical implications. Perhaps this approach should be more universally adopted, so that the limitations of non–statistically significant findings can be addressed and analyzed. Thus, when a different group of investigators analyzed pattern recognition of benign-appearing thyroid nodules for the purpose of suggesting advantages of imaging surveillance over intervention [8], the authors were on terra firma when they concluded, “Use of this approach may substantially [emphasis added] decrease the number of unnecessary biopsy procedures.”

The authors resorted to the use of adverbs to advance their position, but in a more defensible manner because, unlike the words unequivocally and undoubtedly, the reader is invited to consider alternative approaches.

One might consider this current critique as being picayune, trivial, and perhaps rhetorical. The uses of ill-chosen words (like ill-chosen descriptors of a lesion leading to the wrong diagnosis), however, have consequence [9]. For example, in commenting on the nature of future radiology reporting, a published letter to the editor cited and relied on an article in which legal commentary presented conclusionary statements regarding a radiologist's duty to report results directly to a patient, citing important state court decisions supporting this position [10] but omitting other state court decisions that contradict the same position [11]. This initial letter [12], predicated on an unbalanced commentary, generated yet another author's letter to the editor [13], compounding the problem of opinions based on a dubious platform of accuracy. Such a stream of events illustrates the derivative consequences not only to what is printed in the literature but also the manner in which it is presented and interpreted.

Omissions in Reporting 

The publication of the article “Can Peer Review Contribute to Earlier Detection of Breast Cancer? A Quality Initiative to Learn From False-Negative Mammograms” [14] prompted an invited editorial comment by this author in a different journal [15]. The editorial comment clarified that the determination made by the authors was focused on an internal consensus determination of interpretative mistakes. Although errors in interpretation are an important parameter to consider, they are not tantamount to but rather represent only one component of “false-negative mammograms.” The conclusion of the editorial was that the title of the article was both incorrect and misleading [15]. Likewise, in an article promoting the use of nuclear imaging for breast lesions, the abstract indicates the smallest invasive cancer detected as being 1 mm, omitting the more important observation, especially for those limiting their reviews of a given article to the abstract and conclusion [2], that the more relevant mean size of such lesions found was 20 mm, a number found only in the text of the article [16]. Furthermore, in promoting the technology, issues of total body radiation dose compared with alternative approaches were not addressed, though such dose considerations are far from trivial.

Lest one contend that this commentator is singularly critical, consider an editorial comment by an established academic member of the breast community [17] regarding a different article related to infrared thermal imaging [18]:

The abstract of this article could potentially mislead the reader, as it mentions only the best results rather than the full range of observed results and then does not substantiate the proposed conclusion. This article provides an excellent example of why readers need to critically evaluate an entire article instead of relying on the summary information provided in the abstract.

In expressing concern about the information conveyed by this “peer-reviewed” study, the commentator again advances concern that readers may be more dependent on editorial protections that do not successfully condition the presentation of information that is published. Scrutiny of an entire article is always preferable, but it may be predictable folly to believe that many readers with limited time and background in experimental designs will review the methodology. The marketplace of ideas in peer-reviewed published literature is not a venue that should rely on the principle “caveat emptor,” any more than users of medical devices or pharmaceuticals should need to examine, beyond customary caution regarding individual risks, the data that federal regulations are meant to safeguard.

Finally, even nonmedical data are sometimes reported with a “spin” that may facilitate publication but may not serve the objectivity that readers deserve. In describing the reporting of important test results by e-mail, the authors emphasized that 74.6% of the alerts were reviewed by the referring physicians [19]. They opined that their proposed system is a valuable alternative means of communication, even though the direct communication of urgent results is required by both the courts [20] as well as the guidelines on communication of the ACR [21]. While cautioning that backup systems may be required, the authors indicated that they did not follow up on those cases that were not reviewed by the clinician, which constituted more than one-quarter of all results. The failure to act on critical results invites legal redress when direct communication is not involved; omitting the analysis of this group of cases is a shortcoming that deserves greater comment than simply suggesting that backup systems may be needed.

Disclosure 

In 2005, as reports regarding the use of computer-assisted detection in mammography began to increase, >50% of abstracts accepted by the Radiological Society of North America were submitted by teams of investigators, at least one of whom had an equity interest in the technology, as indicated by required disclosures. Financial disclosures have been used as the simplest safeguard for conditioning potential bias, even though it is sometimes not enforced.

Companies that seek to facilitate clinical research, especially underfunded new projects, may offer incentives such as equity to enlist the kind of talent needed for development. To the extent that a quid pro quo effort by both parties advances scientific knowledge, this formula is not inherently misguided. When such investigators lead investigational studies and report results to the exclusion of more arms length investigators with no financial interest in the success of the company, there is an inevitable potential for abuse, sometimes manifested in the kinds of issues discussed throughout this essay. Rather, after initial development, such studies should be corroborated by others so that bias, though not entirely avoidable by virtue of the relationships of the investigators and vendors, is minimized. When an early multisite study was constructed to evaluate the outcome of stereotactic biopsy, the FDA-approved vendor equipment was reduced in price to the institution in return for data collection, a common cooperative relationship that is used by many academic centers. However, the investigators had no financial interest in the impact of the final results [22].

Outlook and Suggetions 

As mentioned in the introduction, the direct and derivative consequences of bias in reporting of investigational studies are no longer a simple academic faux pas. A recent review of data used to support guidelines for use of metformin found that conclusions may not have been validly based on the cited studies [23]. The American Recovery and Reinvestment Act that was recently passed provides for enabling legislation for the Institute of Medicine to promote comparative effectiveness research, much of which will involve medical imaging [24]. The intent of such research should provide a basis for cost control and priorities for reimbursement. If the medical literature suffers from compromised reporting, then the results of such comparisons may lead to misguided recommendations.

Those who volunteer their knowledge, experience, expertise, and time to provide peer review for publications form an invaluable and indispensible resource that has provided an important service to the medical community and public at large. Although reviewers may decline invitations to review, those who accept such assignments should still be cognizant of their own limitations and alert editors that some issues need to be pursued further by others. The process of review and “turnaround” time is burdensome, but such exceptions may prove more valuable than the additional resources required. Consider the initially reported results of the ACRIN® Digital Mammographic Imaging Screening Trial (DMIST). The data initially published in the New England Journal of Medicine, which is read by a broad cross-section of the referring physician population, were revisited by peer reviewers from a radiology journal, reviewers who seemed more versed in this kind of imaging data analysis. The subsequent report was at variance with the initial one. Same data, different pool of peer reviewers, different conclusions [25, 26]. If one considers that reimbursement schedules are predicated on outcome analyses, then the consequences of different results are substantive.

Variation of opinion regarding data is expected and is manifest in the manner in which different individuals review an article submitted for publication. However, certain principles are easy to apply and should help obviate many of the issues raised in this discussion. Unless substantiated, terms that preclude different approaches—what I refer to as “closeout words,” such as unequivocally or always—should not be used in scientific reporting. Reviewers should be encouraged—perhaps reminded—by editors to look for basic and balanced information contained in the abstract and title that properly reflects the content of the study. The use of adverbs and to a lesser extent adjectives should be restricted to those that are essential to communicate an idea without enthusiasm, either positive or negative. The use of conditioning adverbs and adjectives is not inherently ill advised, as illustrated in the previously described report that used the word substantially [8]. Rather, their use in urging a position that is not justified by the data should be discouraged. Indeed, a reviewer of this manuscript noted 7 adjectives and 1 adverb (missing another adverb, recently); the reader is invited to decide if such qualifiers in this essay have been used with balance or in an advocating manner.

In fact, some of these suggested efforts can be supplemented by editorial offices if reviewers have overlooked such infractions. Just as in the elimination of the word significant for description rather than mathematical denotation, words that end in “ly” can be “red flags” for editorial staffs. Editors, while seeking feedback from reviewers, should not permit the latter to make editorial decisions without sufficient basis for the recommendation. Rather, reviewers are called on to apply requisite expertise in helping editors analyze the strength and relevance of the data.

Bias will continue, and future commentaries will remind us of the shortcomings of such bias. More than a decade ago, the Style Manual Committee of the Council of Biology Editors proposed that scientific writing be “accurate, clear, economical, fluent, and graceful” [27]. As economic conditions in health care combined with increasing evidence-based data permit deliberate strategies for health care, bias in reporting will no longer be relegated to academic disputes but rather will have increasing importance and impact on the allocation of resources for clinical practice. Undoubtably!