Tag Archives: Statistical significance

Anti-fluoridation campaigners often use statistical significance to confirm bias

I was pleased to read this Nature article – Five ways to fix statistics – recently as it mirrors my concern at the way statistical analysis is sometimes used to justify or confirm a bias and not reveal a real causal relationship. Frankly these days I just get turned off by media reports of studies showing statistically significant relationships as evidence for or against the latest health or other fads.

As the Nature article says, statistical significance tests often amount “to uncertainty laundering:”

“Any study, no matter how poorly designed and conducted, can lead to statistical significance and thus a declaration of truth or falsity. NHST [null hypothesis significance testing] was supposed to protect researchers from over-interpreting noisy data. Now it has the opposite effect.”

No matter how good a relationship appears, or how significant the statistical analysis shows it to be, it is simply a relationship and may have no mechanistic or causal backing.  An example often used to illustrate this is the close relationship between the prevalence of autism and sales of organic produce.

Clearly statically significant but we don’t find those activists claiming autism is related to one thing or another ever citing this one. I am picking these activists may well have a bias towards organic produce.

Here are several examples I have discussed before which illustrates how “statistical significance” is sometimes used to confirm bias in fluoridation studies. I think these are very relevant as anti-fluoridation campaigners often cite statistical significance as if it is the final proof for their claims.

Ignoring relevant confounders

This is an easy trap for the biased researcher (and let’s face it, most of us are biased – it’s only human). Just ignore other confounders or risk-modifying factors that may be more important. Or ignore the fact that the risk-modifying factor one is interested in (in this case fluoride) may just be acting as a proxy for (and therefore is related to) something else which is more relevant.

This why all credible risk-modifying factors should be considered in correlation studies. They should be included in the statistical analyses.

It’s amazing how many researchers either ignore the possible risk-modifying factors besides their pet one – or pay lip-service to the problem by limiting their consideration to only a small range of such factors.

Examples of studies promoted by anti-fluoride campaigners where this is a problem include:

Peckham et al., (2015) hypothyroidism paper:

Peckham, S., Lowery, D., & Spencer, S. (2015). Are fluoride levels in drinking water associated with hypothyroidism prevalence in England? A large observational study of GP practice data and fluoride levels in drinking water. J Epidemiol Community Health, 1–6.

This has been widely condemned for a number of reasons – one of which is that iodine deficiency, a known factor in hypothyroidism, was not included in the statistical analysis.

(See Paper claiming water fluoridation linked to hypothyroidism slammed by experts and Anti-fluoride hypothyroidism paper slammed yet again).

The  Takahashi et al., (2001) cancer paper:

Takahashi, K., Akiniwa, K., & Narita, K. (2001). Regression Analysis of Cancer Rates and Water Fluoride in the USA based Incidence on IACR / IARC ( WHO ) Data ( 1978-1992 ). Journal of Epidemiology, 11(4), 170–179.

These authors reported an association between fluoridation and a range of cancers. Problem is, they did not consider any other risk-modifying factors. When some geographical parameters were included in the statistical analyses there were no statistically significant relationships of cancer with fluoridation.

(see Fluoridation and cancer).

The Malin & Till (2015) ADHD paper:

Malin, A. J., & Till, C. (2015). Exposure to fluoridated water and attention deficit hyperactivity disorder prevalence among children and adolescents in the United States: an ecological association. Environmental Health, 14.

This reported an association of ADHD prevalence with the extent of fluoridation in the US. Anti-fluoride campaigners have cited this paper a lot because it is the only study indicating any effect of fluoridation on cognitive ability. All other studies they rely on were from areas of endemic fluorosis where the natural levels of fluoride are higher than that used in community water fluoridation.

Malin & Till (2015) considered only household income as a possible risk-modifying factor. No consideration was given to residential elevation which other researchers had around the same time reported as associated with ADHD prevalence.

I repeated their statistical analysis but included residential elevation and a range of other risk-modifying factors. This showed there was no statically signficant association of ADHD with fluoridation when other risk-modifying factors, particularly elevation, were included. My critique of Malin and Till (20215) is now published:

Perrott, K. W. (2017). Fluoridation and attention deficit hyperactivity disorder – a critique of Malin and Till ( 2015 ). Br Dent J.

(See ADHD linked to elevation not fluoridationADHD link to fluoridation claim undermined again and Fluoridation not associated with ADHD – a myth put to rest).

Ignoring the lack of explanatory power

I think this is where the over-reliance on statistical significance, the p-value, can be really misleading. Researchers desperately wishing to confirm their bias will proudly claim  a statistically significant relationship, a p-value less than 0.05, etc., as if that is the final “proof.” These researchers will often hide the real meaning of their relationship by not making the actual data available or limiting their report of their statistical analysis to p-vlaues and, maybe, a mathematical relationship.

However, if the reported relationship actually explains only a small part of the observed variation in the data it may be meaningless. Concentration on such a relationship means that other more signficant risk-modifying factors which would explain more of the variation are ignored. Anyway, where a factor explains only a small part of the variation it is likely a more complete statistical analysis would show that its contribution was not actually statistically signficant.

Some examples:

The prenatal fluoride exposure and IQ study of Bashash et al (2017):

Bashash, M., Thomas, D., Hu, H., Martinez-mier, E. A., Sanchez, B. N., Basu, N., … Hernández-avila, M. (2016). Prenatal Fluoride Exposure and Cognitive Outcomes in Children at 4 and 6 – 12 Years of Age in Mexico.Environmental Health Perspectives, 1, 1–12.

These authors reported a statistically significant association of Child IQ with the prenatal fluoride exposure of their mothers. However, their figures showed a very wide scatter in the data indicating very little explanation of the variation in child IQ by the association with prenatal fluoride. (see below left). This must be why the Fluoride Action Network removed the data points from the figure when reproducing it for their promotion of the paper (see below right).

Bashash et al., (29017) did not give the complete statistical analysis of their data. However, I was able to digitally extract the data from their figure and my analysis showed that prenatal fluoride expose was only able to explain a little over 3% of the variation in child IQ. So, despite the statistical significance of their observed relationship prenatal fluoride exposure is unlikely to be a real factor in child IQ. In fact, concentration on this minor (even if statistically significant) factor will only inhibit the discovery of the real causes of IQ variation in these children.

Yes, anti-fluoride campaigners will protest that this study did consider some other possible risk-modifying factors. However the very low-level of explanation of the variation in the data indicates they did not consider enough.

(see Premature births a factor in cognitive deficits observed in areas of endemic fluorosis? Fluoride, pregnancy and the IQ of offspring and Maternal urinary fluoride/IQ study – an update).

The Xiang et al., (2003) water fluoride and IQ study:

Xiang, Q; Liang, Y; Chen, L; Wang, C; Chen, B; Chen, X; Zhouc, M. (2003). Effect of fluoride in drinking water on children’s intelligence. Fluoride, 36(2), 84–94.

Anti-fluoride campaigners rely a lot on this and other papers from this group.  Even though this research involved areas of endemic fluorosis it, in a sense, provides some of their best evidence because they reported a dose-dependent relationship of IQ to water F. Xiang et al., (2003) claimed a statistically signficant association of child IQ to fluoride water levels.  Other anti-fluoride campaigners, and some other researchers, have cited Xiang et al., (2003) to support such an association.

I don’t question these researchers found a significant association – but there is a problem. Nowhere do they give a statistical analysis or the data to support their claim! Very frustrating for critical readers (and we should all be critical readers).

They did, however, give some evidence from a statical analysis of the relationship of IQ with urinary fluoride. They did not give a complete statistical analysis but they included the data in a figure  (see below) – so I did my own statistical analysis of data digitally extracted from the figure.

The figure shows a high scatter of data points so this is another case of a statistically significant relationship explaining only a small part of the variability. My analysis indicates the relationship explains only about 3% of the variability in IQ value. Another case where researchers have concentrated on their own pet relationship and in the process not properly searched for more reasonable risk-modifying factors capable of explaining a larger proportion of the variation.

I have made a more detailed critique of Xiang et al.  (2015) and Hirzy et al., (2016) which relies on this data (see Does drinking water fluoride influence IQ? A critique of Hirzy et al. (2016)). A paper based on this has been submitted to a journal for publication and is currently undergoing peer review..

(see Anti-fluoride authors indulge in data manipulation and statistical porkiesDebunking a “classic” fluoride-IQ paper by leading anti-fluoride propagandists,  Connett fiddles the data on fluorideConnett & Hirzy do a shonky risk assesment for fluoride and Connett misrepresents the fluoride and IQ data yet again).

Conclusion

This  briefly outlines the statistical problems of a number of papers anti-fluoride campaigners rely on. Two common problems are:

  • Insufficient consideration of confounders or other risk-modifying factors – indicating a bias towards a “preferred” cause, and
  • Reliance on a relationship that, although statistically significant, explains only a very small fraction of the observed variation – again indicating bias towards a “preferred” cause

I don’t for a minute suggest that only those researchers publishing “anti-fluoride” research are guilty of these errors. They are probably quite common. Authors will generally responsibly warn that “correlation does not prove causation” and suggest more work needs to be done including  consideration of a wider number of confounders or risk-modifying factors. However, bias is only human so researcher advocacy for their own findings is understandable. The published research may even be of general value if readers interpret it critically and intelligently.

However, in the political world such critical consideration is very rare. Activists will use published research in the way a drunk uses a lamppost – more for support than for illumination. This makes it important that the rest of us be more objective and critically assess the claims they are making. Part of this critical assessment must include an objective consideration of the published research that is being cited.

Similar articles

Another local climate change denial meme

I commented in Painted into a corner? that the release of NIWA’s report of the recalculation of New Zealand’s 7-station  long term  temperature series had called the local climate change deniers’ bluff. It was time for them to apologise, withdraw their slanderous attacks on NIWA and move on.

Yeah, right! That may have been the sensible thing to do, but of course they are in it for the politics, not the science. So the attacks continue.

Gareth at Hot Topic has an excellent summary of the action (see A Christmas cracker for the cranks). And of course the denier trolls have descended on his post with quite hilarious arguments. One is their claim “this new series shows no warming has occurred here since about 1960.”

Well, of course the series shows nothing of the sort. They appear to basing this incorrect claim on a paragraph in the report which says:

“The unusually steep warming in the 1940-1960 period is paralleled by an unusually large increase in northerly [air] flow during this same period. On a longer timeframe, there has been a trend towards less northerly flow (more southerly) since about 1960. However, New Zealand temperatures have continued to increase over this time, albeit at a reduced rate compared with earlier in the 20th century. This is consistent with a warming of the whole region of the southwest Pacific within which New Zealand is situated.”

Never the less these trolls and their public spokespersons, Richard Treadgold and Brian Leyland are working hardto push this meme. And they claim support from their “eyeball” analysis of the graph in NIWA’s report.

But all the data is available (Seven station data and 11 station data) and anyone can do their own analysis to check these claims.

Continue reading

Deniers distort Phil Jones

Let’s differentiate again between climate change sceptics (who are prepared to rationally consider and discuss the science) and climate change deniers (who have an emotional commitment to denying climate change and see the science as just the result of a huge conspiracy). Currently there is a feeding frenzy amongst the deniers. Hysterical claims are being made and promoted on the flimsiest of evidence, even against the evidence. They are promoting anything and everything that can possibly be misinterpreted to discredit scientific conclusions and scientists. And, yes, these attacks do get nasty and personal.

This must be making the true climate change sceptics very uncomfortable. More of them should publicly dissociate themselves from this hysterical anti-scientific campaign.

Warming since 1995?

Predictably, several local blogging deniers took on the misreporting of the recent BBC interview with climate scientist Phil Jones. Several of his statements have been used to make distorted, even completely fraudulent, claims. Here I only consider the misreporting of Jone’s comment on global warming since 1995:

Here is the question and answer (see Q&A: Professor Phil Jones for full interview):

Question: Do you agree that from 1995 to the present there has been no statistically-significant global warming

Answer: Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level. Achieving statistical significance in scientific terms is much more likely for longer periods, and much less likely for shorter periods.”

At least three local bloggers reported this as a u-turn. A concession that warming was no longer occurring. A denial of previous work! Essentially they were repeated the headlines of overseas conservaqtive and denier newspapers and blogs.

Now you don’t have to be a rocket scientist to see these bloggers are completely distorting the whole meaning of Phil Jones’ comment. But, if you want to dig a bit deeper and get your head around the implications of statistical comments like this watch this video by potholer54. It deals specifically with this distortion:

YouTube – 8a. Climate Change – supplement.

An earlier video from potholer54 deals with the more general question. See  8. Climate Change — Has the Earth been cooling?

Permalink

Similar articles

Reblog this post [with Zemanta]

Share

Climate change confusion – a conspiracy of sorts

The current “climategate” attacks on science must bewilder many scientists. Why the hysteria? Why the wild claims? The facts don’t warrant that sort of reaction!

One is tempted to see a conspiracy – and no doubt there are all sorts of links between organisations and campaigns are easily plotted. But I think what we have is something a bit more uncoordinated. More an accident of history.

Continue reading