I was pleased to read this Nature article – Five ways to fix statistics – recently as it mirrors my concern at the way statistical analysis is sometimes used to justify or confirm a bias and not reveal a real causal relationship. Frankly these days I just get turned off by media reports of studies showing statistically significant relationships as evidence for or against the latest health or other fads.
As the Nature article says, statistical significance tests often amount “to uncertainty laundering:”
“Any study, no matter how poorly designed and conducted, can lead to statistical significance and thus a declaration of truth or falsity. NHST [null hypothesis significance testing] was supposed to protect researchers from over-interpreting noisy data. Now it has the opposite effect.”
No matter how good a relationship appears, or how significant the statistical analysis shows it to be, it is simply a relationship and may have no mechanistic or causal backing. An example often used to illustrate this is the close relationship between the prevalence of autism and sales of organic produce.
Clearly statically significant but we don’t find those activists claiming autism is related to one thing or another ever citing this one. I am picking these activists may well have a bias towards organic produce.
Here are several examples I have discussed before which illustrates how “statistical significance” is sometimes used to confirm bias in fluoridation studies. I think these are very relevant as anti-fluoridation campaigners often cite statistical significance as if it is the final proof for their claims.
Ignoring relevant confounders
This is an easy trap for the biased researcher (and let’s face it, most of us are biased – it’s only human). Just ignore other confounders or risk-modifying factors that may be more important. Or ignore the fact that the risk-modifying factor one is interested in (in this case fluoride) may just be acting as a proxy for (and therefore is related to) something else which is more relevant.
This why all credible risk-modifying factors should be considered in correlation studies. They should be included in the statistical analyses.
It’s amazing how many researchers either ignore the possible risk-modifying factors besides their pet one – or pay lip-service to the problem by limiting their consideration to only a small range of such factors.
Examples of studies promoted by anti-fluoride campaigners where this is a problem include:
Peckham et al., (2015) hypothyroidism paper:
Peckham, S., Lowery, D., & Spencer, S. (2015). Are fluoride levels in drinking water associated with hypothyroidism prevalence in England? A large observational study of GP practice data and fluoride levels in drinking water. J Epidemiol Community Health, 1–6.
This has been widely condemned for a number of reasons – one of which is that iodine deficiency, a known factor in hypothyroidism, was not included in the statistical analysis.
(See Paper claiming water fluoridation linked to hypothyroidism slammed by experts and Anti-fluoride hypothyroidism paper slammed yet again).
The Takahashi et al., (2001) cancer paper:
Takahashi, K., Akiniwa, K., & Narita, K. (2001). Regression Analysis of Cancer Rates and Water Fluoride in the USA based Incidence on IACR / IARC ( WHO ) Data ( 1978-1992 ). Journal of Epidemiology, 11(4), 170–179.
These authors reported an association between fluoridation and a range of cancers. Problem is, they did not consider any other risk-modifying factors. When some geographical parameters were included in the statistical analyses there were no statistically significant relationships of cancer with fluoridation.
(see Fluoridation and cancer).
The Malin & Till (2015) ADHD paper:
Malin, A. J., & Till, C. (2015). Exposure to fluoridated water and attention deficit hyperactivity disorder prevalence among children and adolescents in the United States: an ecological association. Environmental Health, 14.
This reported an association of ADHD prevalence with the extent of fluoridation in the US. Anti-fluoride campaigners have cited this paper a lot because it is the only study indicating any effect of fluoridation on cognitive ability. All other studies they rely on were from areas of endemic fluorosis where the natural levels of fluoride are higher than that used in community water fluoridation.
Malin & Till (2015) considered only household income as a possible risk-modifying factor. No consideration was given to residential elevation which other researchers had around the same time reported as associated with ADHD prevalence.
I repeated their statistical analysis but included residential elevation and a range of other risk-modifying factors. This showed there was no statically signficant association of ADHD with fluoridation when other risk-modifying factors, particularly elevation, were included. My critique of Malin and Till (20215) is now published:
Perrott, K. W. (2017). Fluoridation and attention deficit hyperactivity disorder – a critique of Malin and Till ( 2015 ). Br Dent J.
(See ADHD linked to elevation not fluoridation, ADHD link to fluoridation claim undermined again and Fluoridation not associated with ADHD – a myth put to rest).
Ignoring the lack of explanatory power
I think this is where the over-reliance on statistical significance, the p-value, can be really misleading. Researchers desperately wishing to confirm their bias will proudly claim a statistically significant relationship, a p-value less than 0.05, etc., as if that is the final “proof.” These researchers will often hide the real meaning of their relationship by not making the actual data available or limiting their report of their statistical analysis to p-vlaues and, maybe, a mathematical relationship.
However, if the reported relationship actually explains only a small part of the observed variation in the data it may be meaningless. Concentration on such a relationship means that other more signficant risk-modifying factors which would explain more of the variation are ignored. Anyway, where a factor explains only a small part of the variation it is likely a more complete statistical analysis would show that its contribution was not actually statistically signficant.
Some examples:
The prenatal fluoride exposure and IQ study of Bashash et al (2017):
Bashash, M., Thomas, D., Hu, H., Martinez-mier, E. A., Sanchez, B. N., Basu, N., … Hernández-avila, M. (2016). Prenatal Fluoride Exposure and Cognitive Outcomes in Children at 4 and 6 – 12 Years of Age in Mexico.Environmental Health Perspectives, 1, 1–12.
These authors reported a statistically significant association of Child IQ with the prenatal fluoride exposure of their mothers. However, their figures showed a very wide scatter in the data indicating very little explanation of the variation in child IQ by the association with prenatal fluoride. (see below left). This must be why the Fluoride Action Network removed the data points from the figure when reproducing it for their promotion of the paper (see below right).
Bashash et al., (29017) did not give the complete statistical analysis of their data. However, I was able to digitally extract the data from their figure and my analysis showed that prenatal fluoride expose was only able to explain a little over 3% of the variation in child IQ. So, despite the statistical significance of their observed relationship prenatal fluoride exposure is unlikely to be a real factor in child IQ. In fact, concentration on this minor (even if statistically significant) factor will only inhibit the discovery of the real causes of IQ variation in these children.
Yes, anti-fluoride campaigners will protest that this study did consider some other possible risk-modifying factors. However the very low-level of explanation of the variation in the data indicates they did not consider enough.
(see Premature births a factor in cognitive deficits observed in areas of endemic fluorosis?, Fluoride, pregnancy and the IQ of offspring and Maternal urinary fluoride/IQ study – an update).
The Xiang et al., (2003) water fluoride and IQ study:
Xiang, Q; Liang, Y; Chen, L; Wang, C; Chen, B; Chen, X; Zhouc, M. (2003). Effect of fluoride in drinking water on children’s intelligence. Fluoride, 36(2), 84–94.
Anti-fluoride campaigners rely a lot on this and other papers from this group. Even though this research involved areas of endemic fluorosis it, in a sense, provides some of their best evidence because they reported a dose-dependent relationship of IQ to water F. Xiang et al., (2003) claimed a statistically signficant association of child IQ to fluoride water levels. Other anti-fluoride campaigners, and some other researchers, have cited Xiang et al., (2003) to support such an association.
I don’t question these researchers found a significant association – but there is a problem. Nowhere do they give a statistical analysis or the data to support their claim! Very frustrating for critical readers (and we should all be critical readers).
They did, however, give some evidence from a statical analysis of the relationship of IQ with urinary fluoride. They did not give a complete statistical analysis but they included the data in a figure (see below) – so I did my own statistical analysis of data digitally extracted from the figure.
The figure shows a high scatter of data points so this is another case of a statistically significant relationship explaining only a small part of the variability. My analysis indicates the relationship explains only about 3% of the variability in IQ value. Another case where researchers have concentrated on their own pet relationship and in the process not properly searched for more reasonable risk-modifying factors capable of explaining a larger proportion of the variation.
I have made a more detailed critique of Xiang et al. (2015) and Hirzy et al., (2016) which relies on this data (see Does drinking water fluoride influence IQ? A critique of Hirzy et al. (2016)). A paper based on this has been submitted to a journal for publication and is currently undergoing peer review..
(see Anti-fluoride authors indulge in data manipulation and statistical porkies, Debunking a “classic” fluoride-IQ paper by leading anti-fluoride propagandists, Connett fiddles the data on fluoride, Connett & Hirzy do a shonky risk assesment for fluoride and Connett misrepresents the fluoride and IQ data yet again).
Conclusion
This briefly outlines the statistical problems of a number of papers anti-fluoride campaigners rely on. Two common problems are:
- Insufficient consideration of confounders or other risk-modifying factors – indicating a bias towards a “preferred” cause, and
- Reliance on a relationship that, although statistically significant, explains only a very small fraction of the observed variation – again indicating bias towards a “preferred” cause
I don’t for a minute suggest that only those researchers publishing “anti-fluoride” research are guilty of these errors. They are probably quite common. Authors will generally responsibly warn that “correlation does not prove causation” and suggest more work needs to be done including consideration of a wider number of confounders or risk-modifying factors. However, bias is only human so researcher advocacy for their own findings is understandable. The published research may even be of general value if readers interpret it critically and intelligently.
However, in the political world such critical consideration is very rare. Activists will use published research in the way a drunk uses a lamppost – more for support than for illumination. This makes it important that the rest of us be more objective and critically assess the claims they are making. Part of this critical assessment must include an objective consideration of the published research that is being cited.
Open Parachute 
Excellent analysis Ken.
LikeLike