Flaw and porkie in anti-fluoride report claiming a flaw in Canadian study

Anti-fluoride group, Fluoride Action Network, ironically stamps their own critique of the Calgary fluoridation cessation study as “debunked.”

Anti-fluoride campaigners have launched another attack on a Canadian fluoridation cessation study. They claim it is flawed – but there is a huge flaw in their own critique.

I discussed their original attack in February last year (see Anti-fluoridationist’s flawed attacks on Calgary study). But this new attack is based on a published critique of the original study. I think that is good progress – the anti-fluoride campaigners have made a detailed critique and published it in the journal which published the original paper. The original authors have then responded. This is how things should be done.

Timeline

For those of you with the interest and time who want to go into the details, the original study was published in:

McLaren L, Patterson S, Thawer S, Faris P, McNeil D, Potestio M, Shwart L. (2016) Measuring the short-term impact of fluoridation cessation on dental caries in Grade 2 children using tooth surface indices. Community Dent Oral Epidemiol 2016.

The anti-fluoride critique was recently published in:

Neurath, C., Beck, J. S., Limeback, H., Sprules, W. G., Connett, M., Osmunson, B., & Davis, D. R. (2017). Limitations of fluoridation effectiveness studies: Lessons from Alberta, Canada. Community Dentistry and Oral Epidemiology, (October 2016), 1–7.

The response from the original authors was then published in:

McLaren, L., Patterson, S., Thawer, S., Faris, P., McNeil, D., & Potestio, M. (2017). Fluoridation cessation: More science from Alberta. Community Dentistry and Oral Epidemiology, (October), 1–3.

Other data which have been used in the critique and which I will use here can be found in:

McLaren, L., McNeil, D. A., Potestio, M., Patterson, S., Thawer, S., Faris, P., … Shwart, L. (2016). Equity in children’s dental caries before and after cessation of community water fluoridation: differential impact by dental insurance status and geographic material deprivation. International Journal for Equity in Health, 15(1), 24.

And:

McLaren, L., Patterson, S., Thawer, S., Faris, P., McNeil, D., Potestioa, M. L., & Shwart. L. (2017). Exploring the short-term impact of community water fluoridation cessation on children’s dental caries: a natural experiment in Alberta, Canada. Public Health, 146, 56–64.

Most of the authors of the critique are listed as members of the Fluoride Action Network (FAN) team and I can understand that FAN would feel proud that their critique was published. However, I feel their press release was rather underhand to imply the original study is:

“seriously flawed science  . . . Citizens should be concerned that their tax dollars have funded this biased work.”

And that the work was funded by state and public bodies:

“whose policy is to promote fluoridation.”

But let’s look at the critique itself – because it has some pretty big flaws itself.

What did the original study find?

My article, Anti-fluoridationist’s flawed attacks on Calgary study describes the details of this study. But briefly, it showed that child tooth decay increased in the Canadian city of Calgary after cessation of fluoridation. It used a comparison fluoridated city (the nearby and similar sized city of Edmonton) – and just as well because tooth decay also increased in that city during that time. However, there was still an increase in tooth decay in Calgary after cessation of fluoridation even after correction for the increase due to other factors apparent in Edmonton.

What did the critique claim?

A number of the criticisms are debatable and relatively minor.

How suitable was Edmonton as a comparison city? Neurath et al., (2017) claim it wasn’t suitable (but did not suggest a better alternative). McLaren et al., (2017) claim there is “no better comparison community for Calgary than Edmonton.”

Confounding – Neurath claims consideration of confounding factors was inadequate. McLaren et al., (2017) refer to extra data in two other papers and describes their consideration of several likely confounding factors like public health programmes and use of sealants. Whether the correct factors or sufficient factors were considered is always a bone of contention between authors and critics and, in the end, available data and funding decides.

Study design – Neurath et al., (2017) argue for randomised controlled trials. McLaren et al. (2017) point out that in studies of social programmes one must go with what exists. They say:

“While we agree with the value of stronger designs, one must be thoughtful about evaluation of public health measures, which by definition are complex and context-dependent. We used the best available
data and design for our circumstances”

Data ignored?

But Neurath et al (2017)’s major criticism is that some important data was ignored. And they claim that when that data is included the conclusions are not valid.

Of course, the FAN authors are stretching things quite a bit. The original study was based on data for tooth surfaces – the decay, extracted and filled tooth surfaces (defs). This was used as it is more sensitive than the tooth data itself – the decayed missing and filled teeth (deft).

Data for defs were only available for the 2004/05 and 2014/15 surveys. Unfortunately, there were no defs data for the pre-cessation period closer to the time of cessation (2011). That is the sort of problem researchers face when dealing with existing surveys and existing social programmes.

But the bright sparks at FAN latched on to the fact there was a survey with deft data in Calgary closer to the cessation time – 2009/2010. The fact that there was no equivalent survey for Edmonton didn’t hold them back – they proceeded to imply the 2009/2010 data had been purposely held back, despite McLaren making clear she could not use that data for Calgary in the absence of similar data for Edmonton. That would have negated the requirement for a comparison city and the existing data surely shows that requirement was very wise.

So Neurath et al., (2017) chose to ignore the obvious requirement for a comparison city and proceeded to argue their case on the Calgary data alone. They argued the study was “fatally flawed” and that “key data [was] omitted.” The argument implied the study was somehow fraudulent and that the authors had hidden the 2009/2010 survey data – despite the fact this data is used in another of their papers!

Neurath et al., (2017) pretend that a comparison city is not really necessary – relying only on the tooth data (deft) for Calgary they argue that as 50% of the increased in tooth decay had occurred between the 2004/05 and 2009/10 surveys then the increases seen after cessation of fluoridation was due to the same trend (see their Figure 1 below). They argued this proved that cessation of fluoridation had no effect. Ignoring completely the Edmonton data.

So, an obvious flaw in their critique – but wait, there is more! They actually go so far as to falsify data.

Falsifying a “correction factor”

Not satisfied with the plots in Figure 1B they found a way to make the data look even worse for McLaren et al. (2015). They came up with a “correction” factor to convert the deft data for 2009/2010 survey into defs data. Here is their Figure 2 using the “converted” deft data

Looks bad, doesn’t it?

However, the trick is in the way the conversion factor is calculated. They “used the ratio of defs to deft in the 2013/2014 survey to make the conversion.” The table below for subset (dmft>0) data they used shows this produces a conversion factor of 2.41 – big enough to dramatically push the 2009/10 data point right up so that it is sitting on the Edmonton “trend line” in their Figure 2 above.

But they could have equally used the ratio of defs/deft in the 2004/2005 survey to make the conversion. That produces a much lower conversion factor of 1.63 – which is not at all consistent with their claim “when we applied this conversion [2.41] to the 2004/2005 Calgary survey, where both deft and defs are known, the calculated defs was very close to the known defs.”

In fact, it may have been more appropriate to take the average conversion factors from the two available surveys. In the figure below I have done this (green data point) and compared this with the use of the conversion factors from the 2004/05 survey (purple data point) and that from the 2004/15 survey used by Neurath et al (yellow data point).

I guess this shows the danger of making these sort of adjustments – especially when there is a bias to confirm. And also that readers should beware of vague assertions of the sort:

“when we applied this conversion [2.41] to the 2004/2005 Calgary survey, where both deft and defs are known, the calculated defs was very close to the known defs.”

Conclusion

The McLaren et al., (2017) study has its limitations, limitations admitted and described by the authors. But, it is the FAN critique of Neurath et al., (2017) rather than the original study, that is fatally flawed. Flawed because of confirmation bias and a porky.

1: They ignored the necessary use of a comparison city and assumed the increase in tooth decay in Calgary was linear over the time between the two surveys McLaren at al used.

2: The use of any correction factor would be questionable but Neurath et al., (20127) clearly used a biased value to suit their argument. Further, they purposely misrepresented their correction factor by implying a similar value would have been obtained from the 2004/2005 survey data. Completely wrong.

Similar articles

Ethics and the doping scandal – a response to Guest Work

Grigory Rodchenkov, Russia’s sacked anti-doping lab director. Considered a criminal in Russia and a “whistle blower” in USA. Credit: Emily Berl for The New York Times

International Olympic Committee (IOC) decisions about participation of athletes from the Russian Federation in the Rio Olympics have brought both criticism and support – from the political as well as the sporting communities.

The issues of sports doping, the responsibilities and actions of sporting bodies and the political context and factors all need discussing. So I am pleased to see a Guest Work blog post at SciBlogs from Ian Culpan discussing the ethical questions involved (see Ethics, Doping the Olympics and Russia).

But I think the article missed important ethical considerations and I do not think the issue can properly be discussed without these. To me the following ethical and legal principles, which Ian did not discuss, are central:

1. Proper testing of claims and evidence;
2. Presumption of innocence until proven guilty;
3. Inadmissibility of collective punishment
4. Avoiding direct or implied political direction in decision-making.

A brief background

The Russian Federation does have a problem with sports doping. It should be in everyone’s interests for this to be dealt with. Interestingly, the Russian national officials and politicians do appear to be cooperating with international sports bodies. They have transferred testing of athletes to non-Russian laboratories. Officials (including the deputy Minister of Sport) implicated by Richard McLaren’s World Anti-Doping Authority (WADA) commissioned report have been suspended pending investigation. The President himself has urged officials not to react defensively but to deal with the problem.

Grigory Rodchenkov, the former head of Moscow’s anti-doping laboratory, was a key figure in the current scandal. He was taking bribes to supply illegal drugs to athletes and (apparently) to enable falsification of test results. When he was sacked and criminal proceeding taken against him he fled to the US. Now treated as a “whistle blower” instead of a criminal he made charges implicating higher officials in the doping scandal. His claims made in a May New York Times article (see Russian Doctor Explains How He Helped Beat Doping Tests at the Sochi Olympics) sparked the decision of the WADA to commission the McLaren report.

Richard McLaren’s report effectively supports Rodchenkov’s claims and found Rodchenkov to be trustworthy. But this appears to be McLaren’s opinion, rather than a conclusion based on testing of claims and evidence. There was no attempt to interview officials in the Russian Federation which is surely required for a proper evaluation. And results of the “forensic testing” commissioned by McLaren (DNA data and testing the methods for removing and replacing seals on sample vials and scratches on the vials) are not even included in the report. We are asked simply to accept his judgment on these.

I agree, the time limit of 57 days may well be to blame but in the absence of presentation of the forensic evidence, relying on the claims of an obvious criminal and lack of any consideration of evidence from Russian officials I think Culpan’s judgment the report “seems to contain irrefutable evidence” is just not valid. To interpret a situation where there had been no opportunity given to refute as meaning the evidence was “irrefutable” is hardly fair. Or ethical.

The reliability of the McLaren report and the information he gathered appears to be unravelling – according to articles in The Australian (unfortunately behind a pay wall but see WADA ‘sexed up’ anti-Russia case, implicated clean athletes – Australian media, citing officials). These claim the president of the Australian Olympic Committee, John Coates, who is also an IOC vice president, wrote to Australia’s Health Minister Susan Ley, saying that the IOC had a “lack of confidence in WADA.” There are also problems with the list of “implicated” Russian athletes not named in McLaren’s report but provided to the sporting federations by McLaren. The Australian cites a senior sports official as saying “We were asked to make a judgment about Russian competitors based on McLaren’s report but without having any of the detail to understand the significance of them being named.”

For Richard McLaren’s description and defence of his work see ‘No time to ask Russia’s opinion, I had enough evidence anyway’ – WADA doping report author to RT

Unwarranted judgments are easily made in the context of the current geopolitical struggle and the resulting information war. They can have consequences which are hardly ethical and I think Richard McLaren himself is concerned about this. He said recently:

“The focus has been completely lost and the discussion is not about the Russian labs and Sochi Olympic Games, which was under the direction of the IOC.”

“But what is going on is a hunt for people supposed to be doping but that was never part of my work, although it is starting to (become) so.’’

“My reporting on the state-based system has turned into a pursuit of individual athletes.’’

This treatment of individual Russian athletes, which was described as being like a “Stalinist witch hunt” by one commentator, is what concerns me. I think this raises ethical issues.

Presumption of innocence

This seems so fundamental to our legal (and ethical) system I just cannot see why critics of the IOC have been so prepared to ignore it – or worse, knowingly violate the principle.

Many Russian athletes who have never had a positive drug test have been denied the opportunity to participate in the Rio Olympics. The criteria applied to other Russian athletes has been much harsher than for other nations with sometimes impossible demands being made to prove a long history of clean test results. While athletes from other nations who have been found guilty in the past of doping and “served their time” in suspension are able to compete this is not the case for Russian athletes.

These clean athletes justifiably ask “Why me?” Russian sports fans may well be thoroughly disappointed by this scandal and particularly with athletes and officials who have found to be guilty of doping. But you can understand they are also angry at the unfairness of such discriminatory and unethical judgments made against their clean athletes.

Collective punishment

Punishing clean athletes for the crimes of those who used doping is simply collective punishment. It brings to mind the actions of Nazi occupiers in Eastern Europe who killed innocent villagers (or in some cases killed entire villages) as collective punishment for the actions of partisans. For the life of me, I cannot see how those critics who believe that the entire Russian Olympic Team should have been punished for the (as yet unproven) crimes of some officials consider they occupy the “moral high ground” as Culpan appears to argue.

Not that collective punishment is anything new when it comes the history of staging important international events by the Russian Federation (and the previous USSR). In the 1980s we saw boycotts of the Moscow and San Francisco Olympics. Attempts at collective punishment of entire nations because of disagreements in the international political arena.

There were attempts to inject political issues into the Sochi Olympics, and even promoting the idea of boycotts,  and who seriously doubts that there will be political attempts to harm, or even prevent, the 2018 World Football Cup in the Russian Federation.

Yet, international sporting and cultural events offer great opportunities to encourage goodwill and understanding between nations. They should not be used as weapons in the geopolitical struggle – because that, in turn, only enhances that struggle and harms peace.

Political motivations can prevent a solution

Fortunately, the IOC avoided a blanket ban on athletes from the Russian Federation, despite coming under political pressure to do so. The consequences of such an unprecedented and radical step may have been unpredictable but include a possible break-up of the Olympic movement. This would not have solved the sports doping problem.

As things stand there is now room for progress in a proper investigation of the charges made by Rodchenkov – particularly those suggesting the involvement of state officials. Such serious charges, made by someone facing criminal action, should not be left as they are without a proper balanced investigation. And this investigation must involve officials and legal bodies from the Russian Federation. It is hardly surprising that McLaren’s report is now being described as unfinished. The Australian articles reported IOC spokesman Mark Adams as saying:

“To have someone who didn’t (commit) a competition doping offence but was counted as such is a very dangerous thing. We encourage a full report by Professor McLaren before we make any full and frank ­decisions.’’”

Surely such a proper investigation will have more chance of eliminating Russian sports doping and corruption than external allegations primarily based on claims made by someone fleeing criminal proceedings.

Finally, we should not allow the current concentration on Russian sports doping to fool us into thinking it is only, or even primarily, a Russian problem. The fact is that sports doping is world-wide and there is plenty of evidence that international sporting bodies themselves are not free from corruption.

I presented the most recent official data from WADA n my article Quantifying the problem of international sports doping. This showed that the proportion of positive doping tests for Russian athletes was just less than average for the whole world. More importantly, there are a number of nations with a higher proportion of positive doping tests than the Russian Federation – including India, France, Belgium, Mexico and Turkey.

Yes, the data was for 2014. It did not (could not) cover the current Russian doping scandal or the McLaren report. But let’s not rely on an unethical presumption of guilt to discredit the data.

Let’s not allow geopolitical differences and prejudices get in the way of battling the sports doping problem.

And let’s not allow such differences to lead us to ignore important ethical principles.

Similar articles