Cochrane fluoridation review. II: “Biased” and poor quality research?

Here again, the language used in the Cochrane review (Water fluoridation for the prevention of dental caries) is very misleading. Especially when cherry-picked and taken out of context. The word “bias” used in the review does not have the meaning an uninformed reader might think.

It does not mean motivated experimental design or selection of data to “prove” a predetermined outcome. Rather it has specific meaning related to common (and usually innocent) problems encountered clinical drug trials. These problems also occur in real-life epidemiological studies and trials of the sort used for evaluating social health measures like community water fluoridation (CWF) but the lack of control in such studies means they are harder to combat.. The Cochrane ideal of randomised double-blinded trials is just not realistic in these situations. As the American Academy of Pediatrics comments in their article on the Cochrane review:

“it would be a logistical nightmare to try creating a public water system that pumps fluoridated water to the first house on the block, delivers non-fluoridated water to the following two houses and then provides fluoridated water to the 4th and final house on that block.”

The review describes the types of “biases”  considered:

“Assessment of risk of bias in included studies
. . . . . The domains assessed for each included study included: sampling, confounding, blinding of outcome assessment, completeness of outcome data, risk of selective outcome reporting and risk of other potential sources of bias. . . . . .  We had identified the following factors as important confounders for the primary and secondary outcomes: sugar consumption/dietary habits, SES, ethnicity and the use of other fluoride sources.”

Unfortunately, all these sorts of “biases” are inevitable to some extent in the real world. Researchers do not always have the budget to include consideration of all confounders, or the degree of control required. There are inevitable gaps in data when families move or withdraw children from schools. Yet it is real-world studies, not idealised laboratory experiments, that give the data and other evidence reviewers and decision-makers must consider. Humans can’t be treated like experimental rats.

What “biases” did the review find

As far as “caries outcome” is concerned the review reports a  “high risk of bias overall,” but this “bias may occur in either direction.” This indicates there is not a motivated selection of data or experimental design to produce a predetermined result as that would show up as a systematic bias.

The major cause of “bias” arose from lack of control of the confounding issues of “sugar consumption/dietary habits, SES, ethnicity and the use of other fluoride
sources.” These are of course important factors which could influence results. In the real world it is difficult to control of variations in dietary intake, although socioeconomic status (SES) and ethnicity can be included in statistical analyses of data. My impression is that this is usually done in more recent studies (which the Cochrane review team had excluded from their review – see Cochrane fluoridation review. I: Most research ignored).

Most studies were at low risk of “bias” from sampling methods but on “detection bias” the reviewers report:

The majority of the studies did not blind outcome assessors. This is perhaps unsurprising when considering the efforts that may be required to blind assessors for this type of study.”

The qualification here surely indicates the inadequacy of the Cochrane criteria for using word’s like “bias” which are more fitted to clinical drug trials than evaluation of social health policies.

Most studies did not suffer from “incomplete outcome data” where some data is not measured, but some showed the “bias” of “selective reporting” where data sets reported were incomplete. Interestingly the reviewers report one study where:

“the baseline fluoridation status of the children was determined by the location of the school they attended, which may not have taken into account any children attending schools in fluoridated areas who resided outside those areas.”

This must be a common problem researchers face when they do such real-world epidemiological studies.

Inappropriate criteria used to judge quality of research

Given the nature of evaluating a social health policy like community water fluoridation (CWF) I think the criteria used by the review team to judge the quality fo available research was quite wrong. Their criteria were more fitted to judging clinical drug trials and not social health policies. They acknowledge this in their discussion section “Quality of evidence:”

“However, there has been much debate around the appropriateness of GRADE when applied to public health interventions, particularly for research questions where evidence from randomised controlled trials is never going to be available due to the unfeasibility of conducting such trials. Community water fluoridation is one such area.”

And:

“we accept that the terminology of ’low quality’ for evidence may appear too judgmental. We acknowledge that studies on water fluoridation, as for many public health interventions, are complex to undertake and that researchers are often constrained in their study design by practical considerations. For many public health interventions, the GRADE framework will always result in a rating of low or very low quality. Decision makers need to recognise that for some areas of research, the quality of the evidence will never be ’high’ and that, as for any intervention, the recommendation for its use depends not just upon the quality of the evidence but also on factors such as acceptability and cost-effectiveness (Burford 2012).” My emphasis.

These are important qualifications which, however, did not make it into the review’s Abstract or Plain Language Summary – and certainly not into media reporting. I think the review team was irresponsible to omit such qualifications from their summaries – and many people might suggest they were irresponsible to use such inappropriate criteria for their judgements in the first place.

The scientific literature is not perfect

I keep stressing that readers should always approach scientific reports and papers critically and intelligently. The problems identified by the Cochrane reviewers are inevitable and should always be taken into account by sensible readers. Simple reliance on the abstract of a paper or report often gives a misleading interpretation of the findings. Unfortunately, even unmotivated reporters tend not to read reports in full. Motivated activists will purposely resort to cherry-picking and distortion.

Decision-makers don’t necessarily need perfect scientific papers as they have to consider far more than the abstract conclusion of a scientific paper. There are the democratically expressed views of the electorate and the real situation where social health policies are put into effect. While the Cochrane reviewers expressed “limited confidence” in the size of the effect of CWF on tooth decay. However, policy-makers are more interested in the fact that there is a positive effect on oral health than the possible “theoretical” size of that effect.After all, policy-makers have to also consider the possible role of confounding effects like alternative sources of fluoride, the quality of dental health in the area, socioeconomic status of the population and school health programmes when making decisions about local CWF programmes.

Conclusions

1: The Cochrane reviewers’ use of terms like “bias” and judgment of studies as being of poor quality is inappropriate for evaluation of a social health policy. According to them:

“The main areas of concern were confounding and lack of blind outcome assessment. The evidence was additionally downgraded for indirectness due to the fact that about 71% of the caries studies that evaluated the initiation of water fluoridation were conducted prior to 1975.”

Yet they qualify this by acknowledging such judgement of “bias” and poor quality is inappropriate for a social health policy. And it was their own criteria for rejecting studies that produced a paucity of more recent studies (see Cochrane fluoridation review. I: Most research ignored).

2: These qualifications were not mentioned in the review’s Abstract or Plain Language Summary. I believe this was irresponsible of the authors. especially  given the controversial nature of the subject and the well-understood fact that media reporters rarely read beyond abstracts and summaries.

Such inappropriate and unqualified language provides a godsend to anti-fluoridation propagandists who are already cherry-picking and misrepresenting the review’s main findings.

3: We can remove the inappropriate and judgmental language and still accept that many of the problems identified in the review are inevitable for studies of social health measures. The review actually acknowledges that.

However, the sensible reader of scientific literature is surely aware of these problems. Any research paper must be assessed intelligently and critically – especially regarding the treatment of confounding factors. This is a point I have continually stressed in my posts on this subject.

In my experience, it has been the confirmation bias of anti-fluoride activists which leads them to ignoring such advice. One need only consider their use of studies related to IQ and fluoride in areas of endemic fluorosis, or their recent promotion of poor quality papers claiming a relationship between CWF and Attention-Deficit Hyperactivity Disorder (see ADHD linked to elevation not fluoridation) or hypothyroidism (see Paper claiming water fluoridation linked to hypothyroidism slammed by experts). 

I urge readers to follow this same advice with the Cochrane review. Don’t accept media reports or a limited reading of its Abstract or Plain Language Summary.

Read the whole review – intelligently and sceptically.
See also:
Misrepresentation of the new Cochrane fluoridation review
Cochrane fluoridation review. I: Most research ignored
Cochrane fluoridation review. III: Misleading section on dental fluorosis

Similar articles