Bias in Scientific Studies

Delimitation

Bias must be distinguished from fraud, oversights, misunderstandings, and nonsense arithmetic. It must further be distinguished from the field of statistical pitfalls, illusions, and paradoxes, though each of these, when unrecognized, may bias perceptions and recommendations.

The classical borderline between random error and bias is sometimes fuzzy. The label “bias” is often used about poor data recording, regardless of whether it will affect conclusions and, if so, how. Moreover, blunt procedures (imprecise measurements) may delay the recognition of a health hazard, or benefit, and in that sense pure randomness is itself “biased” against public interests.

Recognition of Bias

Just as, while there is no checklist for the quality of poems, one can develop one's flair for good poetry, the field of bias is open ended. Notwithstanding attempts, it is impossible to devise an exhaustive list of mutually exclusive bias types. Even broad categories such as selection bias and information bias meet at hazy frontiers. But everybody can train his or her flair for detecting bias.

Overly critical readers sometimes find bias where it isn't (bias bias), or reject investigations on grounds of bias even when the bias is obviously negligible or purely hypothetical.

Texts often explain a bias by means of hypothetical examples from which all unnecessary adornment has been peeled off. This is the strength, not the weakness, of such examples. “Real patients do not look like that!” is an often-heard but invalid objection. Precisely, the complexity of clinical data often lies behind an investigator's failure to realize that his or her research procedure is biased.

The Estimand

One cannot discuss hits and misses without a bull's-eye. So any discussion of bias presupposes a defined target, the estimand. Not until agreement about the estimand has been reached can the statistician and client proceed to discuss bias and, subsequently, random uncertainty. Key questions are as follows: What do we want to measure? What is a rational measure thereof? For example, What is a rational measure of successful rehabilitation after multitrauma? What precisely is meant by “the waiting time for liver transplantation in 2006”?

There are four rules of thumb for establishing the estimand. (1) It should be conceptually well-defined (often by imagining an ideal method being applied to 10,000 truly representative cases). (2) Its definition should be detached from study design ([Page 81]i.e., it should parameterize the object process, not the inspection process, with its potential sources of bias). (3) In predictive settings, prospectivity should be built into the definition. This calls for a notion of “a population of naturally occurring identical-looking instances” (case stream), to which predictions are meant to apply. Anything that requires hindsight should be weeded out. Care must be taken to define the right units of prediction (women vs. pregnancies; bladder tumors vs. control cystos-copies). (4) Biased and data-driven agendas should be avoided. These remarks apply, mutatis mutandis, to qualitative research questions as well.

In studies whose key purpose is comparative, internal validity refers to the comparison being fair and external validity to the comparison's matching an envisaged target population. Generalizability (a broader term) refers to the applicability of study results outside the population sampled.

...