Tag Archives: Cochrane

Cochrane fluoridation review described as “empty”

What-the-Cochrane-Review-Should-Have-Concluded

Image credit: MSoF – Ooops, [Newsweek] Did It Again

Publication of the Cochrane fluoridation review last year caused quite a flurry. Anti-fluoridation activists launched an avalanche of cherry-picked quotations and claims that completely misrepresented the review’s findings. On the other hand, the Cochrane review was poorly written and laid itself open to this cherry picking and misrepresentation.

Now, an authoritative group of authors has published a new critique which summarises the faults of the Cochrane review and shows how it has come to be misrepresented. The paper is:

Rugg-Gunn, A. J., Spencer, A. J., Whelton, H. P., Jones, C., Beal, J. F., Castle, P., … Zusman, S. P. (2016). Critique of the review of ‘Water fluoridation for the prevention of dental caries’ published by the Cochrane Collaboration in 2015. Bdj, 220(7), 335–340.

The on-line version is full text so readers can check it out for themselves.

The authors say:

“The Cochrane Review’s conclusion that ‘there is very little contemporary evidence. that has evaluated the effectiveness of water fluoridation for the prevention of caries’ is self-fulfiling due to its omission of contemporary studies designed for surveillance of public health programmes.

So they say:

“This, it could be argued, led to what is termed ‘an empty review’.”

And this is because a key problem of the review was “the restrictive inclusion criteria  used to judge adequacy of study design and risk of bias.”

Nature of water fluoridation programmes

A problem with the Cochrane review is that its judgement criteria were more suitable for drug trials than for evaluation of a social health policy. Drug evaluation relies on randomised controlled trials (RCTs), but:

“With public health interventions things are different. There will only sometimes be RCTs demonstrating efficacy and effectiveness. There are frequently no such trials because the highly complex practical, ethical and financial factors involved mean that RCTs are not feasible. Consequently, when determining whether a public health intervention is cost effective, evidence has to be drawn from a wide variety of other scientific methods and research designs including cross-sectional ones and process evaluations. . . . .

“In many cases it is simply impossible to make recommendations for public health interventions and policy if reliance is only placed on RCTs. Further, with public health interventions, the issue is not about individual patient benefit but whether the population as a whole will benefit.”

There are also many confounding effects with fluoridation:

“variations in use of other sources of fluoride, mainly from toothpaste, and diet, particularly sugar consumption. In many societies, these are closely linked to SES, and evaluation should also measure and control for these explanatory factors and interactions.”

Therefore:

“Water fluoridation should be evaluated using contemporary methods which are appropriate for evaluating public health interventions with such complexities, and systematic reviews should take this into account.”

The Cochrane review failed to do this and, although these problems were aired in the review’s discussion, the abstract and main conclusions were, therefore, misleading and open to misrepresentation.

Criteria for study inclusion

This “purist” approach was also demonstrated in the review’s criteria for study selection. The Cochrane authors considered:

“only prospective studies with a concurrent control, comparing at least two populations, one receiving fluoridated water and the other non-fluoridated water, with at least two points in time evaluated. Groups had to be comparable in terms of fluoridated water at baseline.”

But baseline data can be irrelevant when considering long-term fluoridation schemes. Rugg-Gunn et al., (2016) point out:

“Over time, in many countries, coverage of the population with water fluoridation schemes was almost complete, at least to the limits of public health requirements and technical feasibility. In such jurisdictions, the priority for health authorities was to monitor the continued effectiveness of existing schemes. Most recent evaluations of water fluoridation have been of this type, using the most appropriate design, which is a single cross-sectional survey of fluoridated and non-fluoridated groups with control for confounding factors.” One of the critical problems with the 2015 Cochrane Review is that these data have been excluded from the Review.

This, although recent evaluations have usually been of higher quality than older ones because of the availability of computer processing, use of more sophisticated statistical analysis and greater awareness of the need to taker confounding factors into account.

The authors discuss the impracticability of requiring baseline data for such long-term schemes. The Cochrane reviewers themselves stipulated baseline data should be taken within 3 years of implementing water fluoridation:

“an acknowledgement that the communities may, mainly through population change, lose comparability after three years. While this assumption of similarity may be reasonable over a short period, it becomes less tenable as the period between baseline and final examinations increases.”

This requirement looks silly when considering recent data from long-term schemes or considering possible benefits for 50-year-olds because:

“baseline information on the caries experience of people of this age would be required in the community to be fluoridated and in a comparable reference community, as well as information to be collected 50 years later on the caries experience of people from the same age group in the same communities which have continued to remain fluoridated or non-fluoridated for the whole of that very long period. Such requirements are unfeasibly stringent given the potential for community demographic characteristics to change over time, and render 50-year historical comparability of intervention and reference communities meaningless for present-day comparisons.[My emphasis]

Rugg-Gunn et al., (2016) discuss in-depth other aspects of the criteria used, exclusions if modern research by the Cochrane authors and study design.

Dental fluorosis

The Cochrane review’s inclusion of dental fluorosis seems to be “tacked on” and does not use the criteria outlined for their review and selection of caries studies. In particular, it ignored the influence of different sources of dietary fluoride – leaving the impression that the resulting data related to water fluoridation when it didn’t:

“In the Cochrane Review, the effect of water fluoridation on the prevalence of fluorosis should have been isolated from the confounding effect of other fluorides. The Cochrane Review’s analysis of fluorosis studies is silent on the possible contribution of other fluorides, such as fluoridated toothpaste, which risks leaving readers with the impression that all dental fluorosis arises from fluoride in water supplies. Research since 2000 has indicated that a greater proportion of dental fluorosis risk is due to the use (and therefore swallowing) of fluoride-containing toothpastes than to optimally fluoridated water.

Consequently, an intelligent interpretation of the fluorosis data in the Cochrane review requires some extra calculation – of the sort the reviewers themselves did for the caries studies:

“This comparison between intervention and reference communities was the method used for evaluating caries prevention in the Cochrane Review but, for an unexplained reason, not for the evaluation of dental fluorosis. For communities with lower fluoride concentrations (such as 0.5 mgF/L), their fluorosis levels should be compared with those in the corresponding reference (non-fluoridated) community.”

And

“The highly restrictive approach taken by the Cochrane Review in examining the effect of community water fluoridation on dental caries seems to have been abandoned for dental fluorosis. The reason for this difference is unclear.”

I attempted this “extra calculation” in my article Cochrane fluoridation review. III: Misleading section on dental fluorosis.”

When differences between “fluoridated” and “non-fluoridated” areas are considered there was no significant contribution of water fluoridation to the “dental fluorosis of aesthetic concern.” However, because  this comparison was not made in the Cochrane review anti-fluoride campaigners are claiming that fluoridation causes a prevalence of 12% “dental fluorosis of aesthetic concern!”

That is very misleading.

Anti-fluoride campaigners love to quote prevalence figures for all forms of dental fluorosis, not just the more serious – implying that even the mildest forms should concern us. Consequently, they cite the Cochrane review to claim a 40% prevalence of dental fluorosis This is for all forms from the most severe to the mildest. But, in fact, when the differences between “fluoridated” and “unfluoridated” areas are calculated the prevalence of all forms of dental fluorosis attributable to community water fluoridation is only 7%. And, remember, these will be only the mildest forms.

Again, very misleading.

Conclusions

The Cochrane fluoridation review agrees with all other authoritative reviews when it states:

“that water fluoridation is effective at reducing caries levels in both deciduous [primary] and permanent dentition in children.”

But its conclusions conflict with the literature on:

“the effectiveness of water fluoridation in respect of: its effectiveness in adults; its effectiveness in reducing social disparities in oral health; and the effect of cessation of water fluoridation. On these, the Cochrane Review said that there was insufficient evidence; it did not say that water fluoridation was ineffective in these regards.”

On these, the review would only say that “there is insufficient evidence.” But that is a self-fulfilling conclusion given their restrictive selection criteria. Some observers may actually say such a conclusion is irresponsible because:

“It is a fundamental premise of interpreting evidence from trials that the absence of evidence, or the existence of poor-quality evidence, should not be confused with, or taken to imply, an absence of effect. There is a risk that the Cochrane Review will be inadvertently, or deliberately, misinterpreted in this way.”

And we know this is, in fact, what has happened. Motivated anti-fluoride campaigners have chosen to present an absence of evidence (because of the restricted selection of studies) as “proof” that fluoridation is not effective.

It just shows how an “empty review” can be used to make the most unwarranted claims.

Similar articles on the Cochrane fluoridation review

Cochrane responds to misrepresentation of their fluoridation review

feedback

Image Credit: Cochrane Oral Health Blog

The latest Cochrane Review on community water fluoridation (CWF) was published in June. Here are a citation and link for those interested:

Iheozor-Ejiofor, Z., Worthington, HV., Walsh, T., O’Malley, L., Clarkson, JE., Macey, R., Alam, R., Tugwell, P., Welch, V., Glenny, A. (2015). Water fluoridation for the prevention of dental caries (Review). The Cochrane Library, (6).

Immediately after publication, anti-fluoride propagandists launched a campaign of misrepresentation and outright distortion of the review’s findings. I dealt with some of this, and commented on the review itself, in the following posts:

The wave of misrepresentation and situations concerned health professionals – some of their on-line feedback and responses are in the Cochrane blog posts – Little contemporary evidence to evaluate effectiveness of fluoride in the water and Our response to the feedback on the Cochrane fluoridation review).

The Cochrane Oral Health Group yesterday published an updated Plain Language Summary (PLS) for the review. If you want to look in detail here is the original version of the review, and here is the abstract and updated Plain Language Summary from the latest version (now online). Their short explanation for this is:

“Following feedback, from a variety of sources, we felt it was necessary to make the language of the PLS simpler.”

This is logical. The PLS is the only part of the Review most policy makers will read. The old version contained too many words like “bias” and references to research “quality” which may have been reasonable to an academic audience but conveyed an entirely different meaning to policy makers who do not have an academic or scientific background. Anti-fluoride campaigners have worked hard to use this in their misrepresentations and distortions aimed at policy makers as well as the public.

Some of the changes

The new PLS does not include the word “bias” and now describes the selection criteria pointing out most studies made after 1975 were excluded (because they did not include initial surveys). Readers will now be more aware that the lack of information in some areas resulted from these strict selection criteria and not from lack of research.

For example, the text:

“No studies met the review’s inclusion criteria that investigated the effectiveness of water fluoridation for preventing tooth decay in adults, rather than children”

has been replaced by

“Within the ‘before and after’ studies we were looking for, we did not find any on the benefits of fluoridated water for adults.”

And the text:

“There was insufficient information available to find out whether the introduction of a water fluoridation programme changed existing differences”

has been replaced by:

“We found insufficient information to determine whether fluoridation reduces differences in tooth decay levels between children from
poorer and more affluent backgrounds.”

Will the misrepresentation continue?

Of course it will. Even the most carefully worded summary can be distorted to misrepresent reported findings. Hopefully, though, these changes will make it harder for campaigners to pull the wool over the eyes of policy makers. The careful reader will now have a better idea of the limitations of the review resulting from the strict selection criteria. Hopefully, they will also be aware that statements like “We found insufficient information . . .” do not mean there is no information. Nor does the inability, within the restricted selection criteria, to find an effect mean there is no effect.

I am disappointed that their changes did not make the situation of dental fluorosis clearer. They do now stress that most of the dental fluorosis studies reviewed “were conducted in places with naturally occurring – not added – fluoride in their water.” But this is not adequate:

“results of the studies reviewed suggest that, where the fluoride level in water is 0.7 ppm, there is a chance of around 12% of people having dental fluorosis that may cause concern about how their teeth look.”

is just not adequate

The choice of 0.7 ppm will be seen as relevant to the concentration used in CWF – but this does not mention that any difference between the  prevalence in fluoridated and unfluoridated areas is very small and not statistically significant. In other words, their comments on dental fluorosis are still not relevant to CWF.

Similar articles

Comparing the Cochrane and NZ Fluoridation Reviews

Sci Rev

New Zealand policy makers and health professionals should be wary about much of the current media comments on the Cochrane Fluoridation Review (Iheozor-Ejiofor 2015). Anti-fluoridation campaigners are misrepresenting it and distorting its findings. They are using cherry-picked quotes to make claims about the review which are just not true.

Some are even claiming (wrongly) that the Cochrane review findings conflict with this in the NZ Fluoridation Review (Eason et al., 2014). Or that, simply because it was published a few months after the NZ Review it somehow makes the NZ Review obsolete.

Review findings agree

Nothing could be further from the truth. The findings in the Cochrane Review do not conflict with those in the NZ  Review. And, because the Cochrane Review is much more limited than the NZ Review, policy makers and health professionals should not consider that as the only document required for their reading.

In particular, the Cochrane Review considered only questions of community water fluoridation (CWF) efficacy. It did not consider aspects related to health concerns which, of course, are always in the front of the minds of policy makers and health professionals.

I have done a side-by-side comparison of the two reviews and summarise their findings below

CWF efficacy

The Cochrane reviewers produced a quantitative estimate for the effect of CWF on dental decay, but only for children and used only studies satisfying their strict selection criteria (see Cochrane fluoridation review. I: Most research ignored). This unfortunately excluded more recent high-quality cross-sectional studies.

The NZ Reviewers did not produce an overall quantitative estimate but made more general conclusions.

Cochrane Review

NZ Fluoridation Review

Efficacy of CWF
“Data suggest that the introduction of water fluoridation resulted in a 35% reduction in decayed, missing or filled baby teeth and a 26% reduction in decayed, missing or filled permanent teeth. It also increased the percentage of children with no decay by 15%. These results indicate that water fluoridation is effective at reducing levels of tooth decay in both children’s baby and permanent teeth”. “Analysis of evidence from a large number of epidemiological studies and thorough systematic reviews has confirmed a beneficial effect of CWF on oral health throughout the lifespan. This includes relatively recent studies in the context of the overall reduced burden of caries that has resulted from the widespread use of topical fluoride products (e.g. toothpastes,  mouth rinses, and fluoride varnishes).”
Adult caries
No conclusions because of study selection limitations. “Although most studies of the effects of CWF have focused on benefits in children, caries
experience continues to accumulate with age, and CWF has also been found to help reduce the extent and severity of dental decay in adults, particularly with prolonged exposure. The long history of CWF around the world now means that many adults in late life have experienced a lifetime of fluoridation. The benefits for adult dental health include lower levels of root caries, and better tooth retention into old age.”
Socio-economic effects
No conclusions because of study selection limitations. “The burden of tooth decay is highest among the most deprived socioeconomic groups, and this is the segment of the population for which the benefits of CWF appear to be greatest. CWF appears to be most cost-effective in those communities that are most in need of improved oral health. In New Zealand, these include communities of low socioeconomic status, and those with a high proportion of children or Māori. A number of studies have suggested that the benefits of CWF are greatest among the most deprived socioeconomic groups, although the magnitude of the difference is uncertain.”
Effect of stopping fluoridation
No conclusions because of study selection limitations. “Stopping CWF leads to ~17% increase in caries experience”  cited from US Task Force on Community Preventive Services
Influence of fluoridated toothpaste, etc.
No conclusions because of study selection limitations. The beneficial effect of CWF on oral health is still shown in relatively recent studies illustrating the overall reduced burden of caries that has resulted from the widespread use of topical fluoride products (e.g. toothpastes,  mouth rinses, and fluoride varnishes). “In New Zealand, significant differences in decay rates between fluoridated and non-fluoridated communities continue to exist, despite the fact that the majority of people use fluoride toothpastes.”

Health issues related to CWF

Dental fluorosis is generally considered the only negative health results of CWF. Both Reviews did consider dental fluorosis, although the Cochrane review did not specifically compare fluoridated and unfluoridated areas – which is necessary to determine the effect of fluoridation on dental fluorosis prevalence. See Cochrane fluoridation review. III: Misleading section on dental fluorosis for a discussion of this and an estimate fo the effect of CWF on dental fluorosis calculated using the Cochrane data.

The Cochrane review did not consider any other health effects.

Cochrane Review

NZ Fluoridation Review

Dental Fluorosis
Only calculated effect of fluoride intake in dental fluorosis. The effect of CWF itself was not considered. However, this can be estimated by subtracting prevalence for unfluoridated region. These estimates indicate that dental fluorosis levels of aesthetic concern are similar in fluoridated and unfluoridated areas (see Cochrane fluoridation review. III: Misleading section on dental fluorosis).
.
“The prevalence of fluorosis of aesthetic concern is minimal in New Zealand, and is
not different between fluoridated and non-fluoridated communities, confirming that a substantial proportion of the risk is attributable to the intake of fluoride from sources other
than water (most notably, the swallowing of high-fluoride toothpaste by young children).
The current fluoridation levels therefore appear to be appropriate. It is important, however, that the chosen limit continues to protect the majority of high-exposure individuals.”
IQ effects
Not considered “We conclude that on the available evidence there is no appreciable effect on cognition arising from CWF.”
Cancer
Not considered “We conclude that on the available evidence there is no appreciable risk of cancer arising from CWF.”
Kidney
Not considered “Studies and systematic reviews have found no evidence that consumption of optimally fluoridated drinking water increases the risk of developing kidney disease. However, individuals with impaired kidney function experience higher/more prolonged fluoride exposure after
ingestion because of reduced urinary fluoride excretion, and those with end stage kidney
disease may be at greater risk of fluorosis.”

Conclusions

The Cochrane review is far more limited in its coverage than the NZ Fluoridation Review. It did not consider possible health effects (apart from dental fluorosis) which is an important aspect of the fluoridation controversy for health professionals and policy makers.

The two Reviews agree that CWF is effective for children, but the NZ Review also considered effectiveness for adults, the reduction of socioeconomic differences in oral health and effects of stopping fluoridation on tooth decay. It also considered more recent research than the Cochrane review, so was able to discuss possible reduction in the efficacy of CWF due to the use of fluoridated toothpaste in recent years.

The Cochrane review does not make the NZ Fluoridation  Review obsolete at all. Nor do its conclusions conflict with those of the New Zealand Review.

Policy makers and health professionals should pay attention to both reviews in making judgements of CWF efficacy, but will need to use the NZ Review for their judgements on possible health effects.

References

New Zealand Fluoridation Review:
Eason, C., & Elwood, JM. Seymour, Thomson, WM. Wilson, N. Prendergast, K. (2014). Health effects of water fluoridation : A review of the scientific evidence

Cochrane Fluoridation Review:
Iheozor-Ejiofor, Z., Worthington, HV., Walsh, T., O’Malley, L., Clarkson, JE., Macey, R., Alam, R., Tugwell, P., Welch, V., Glenny, A. (2015). Water fluoridation for the prevention of dental caries (Review). The Cochrane Library, (6).

Similar articles