The promotion of weak statistical relationships in science

Image credit: Correlation, Causation, and Their Impact on AB Testing

Correlation is never evidence for causation – but, unfortunately, many scientific articles imply that it is. While paying lip service to the correlation-causation mantra, some (possibly many) authors end up arguing that their data is evidence for an effect based solely on the correlations they observe. This is one of the reasons for the replication crisis in science where contradictory results are being reported. Results which cannot be replicated by other workers (see I don’t “believe” in science – and neither should you).

Career prospects, institutional pressure and the need for public recognition will encourage scientists to publish poor quality work that they then use to claim that have found an effect. The problem is that the public, the news media and even many scientists simply do not properly scrutinise the published papers. In most cases they don’t have the specific skills required for this.

There is nothing wrong with doing statistical analyses and producing correlations. However such correlations should be used to suggest future more meaningful and better-designed research like randomised controlled trials (see Smith & Ebrahim 2002 – Data dredging, bias, or confounding. They can all get you into the BMJ and the Friday papers. ). They should never be used as “proof” for an effect, let alone argue that the correlation is evidence to support regulations and advise policymakers.

Hunting for correlations

However, researchers will continue to publish correlations and make great claims for them because they face powerful incentives to promote even unreliable research results. Scientific culture and institutional pressures provide expectations demanding academic researchers produce publishable results. This pressure is so great they will often clutch at straws to produce correlations even when the initial statistical analyst produces none. They will end up “torturing the data.”

These days epidemiological researchers use large databases and powerful statistical software in their search for correlations. Unfortunately, this leads to data mining which, by suitable selection of variables, makes the discovery of statistically significant correlations easy. The data mining approach also means that the often cite p-values are meaningless. P-values measure the probability the relationship occurs by chance and often cited as evidence of the “robustness” of the correlations. But probability is so much greater when researchers resort to checking a range of variables and that isn’t reflected properly in the p-values.

Where data mining occurs, even to a limited extent, researchers are simply attempting to make a purse out of sow’s ear when they support their correlations merely by citing a p-value < 0.05  because these values are meaningless in such cases. The fact that so many of these authors often ignore more meaningful results from their statistical analyses (like R-squared values which indicate the extent that the correlation “explain” the variation in their data) underlines their deceptive approach.

Poor statistical relationships

Consider these correlations below – two data sets are taken from a published paper – the other four use random data provided by Jim Jones in his book Regression Analysis: An Intuitive Guide.

You can probably guess which correlations were from real data (J and M) because there are so many more data points All of these have correlations low p values – but of course, those selected from random data sets resulted from data mining and the p-values are therefore meaningless because they are just a few of the many checked. Remember, a p-value < 0.05 means that the probability of a chance effect is one in twenty and more than twenty variable pairs were checked in this random dataset.

The other two correlations are taken from Bashash et al (2017). They do not give details of how many other variables were checked in the dataset used but it is inevitable that some degree of data mining occurred. So, again, the low p-values are probably meaningless.

J provides the correlation of General Cognitive Index (GCI) scores in children at age 4 years with maternal prenatal urinary fluoride and M provides the correlation of children’s IQ at age 6–12 y with maternal prenatal urinary fluoride. The paper has been heavily promoted by anti-fluoride scientists and activists. None of the promoters have made a critical, objective, analysis of the correlations reported. Paul Connett, director of the Fluoride Action Network, was merely supporting his anti-fluoride activist bias when he uncritically described the correlations as “robust.” They just aren’t.

There is a very high degree of scattering in both these correlations, and the R-squared values indicate they cannot explain any more than about 3 or 4% of the variance in the data. Hardly something to hang one’s hat on, or to be used to argue that policymakers should introduce new regulations controlling community water fluoridation or ban it altogether.

In an effort to make their correlations look better these authors imposed confidence intervals on the graphs (see below). This Xkcd cartoon on curve fitting gives a cynical take on that. The grey areas in the graphs may impress some people but it does not hide the wide scatter of the data points. The confidence intervals refer to estimates of the regression coefficient but when it comes to using the correlations to predict likely effects one must use the prediction intervals which are very large (see Paul Connett’s misrepresentation of maternal F exposure study debunked). In fact, the estimated slopes in these graphs are meaningless when it comes to predictions.

Correlations reported by Bashash et al (2017). The regressions explain very little of the variance in the data and connect be used to make meaningful predictions.

In critiquing the Bashash et al (2017) paper I must concede that at least they made their data available – the data points in the two figures. While they did not provide full or proper results from their statistical analysis (for example they didn’t cite the R-squared values) the data does at least make it possible for other researchers to check their conclusions.

Unfortunately, many authors simply cite p-values and possible confidence intervals for the estimate of the regression coefficient without providing any data or images. This is frustrating for the intelligent scientific reader attempting to critically evaluate their claims.

Conclusions

We should never forget that correlations, no matter how impressive, do not mean causation. It is very poor science to suggest they do.

Nevertheless, many research resort to correlations they have managed to glean from databases, usually resorting to some extent of data mining, to claim they have found an effect and to get published. The drive to publish means that even very poor correlations get promoted and are used by ideologically or career-minded scientists, and by activists, to attempt to convince policymakers of their cause.

Image credit: Xkcd – Correlation

Remember, correlations are never evidence of causation.

Similar articles

Can we trust science?

Image credit: Museum collections as research data

Studies based simply on statistically significant relationships found by mining data from large databases are a big problem in the scientific literature. Problematic because data mining, or worse data dredging, easily produces relationships that are statistically significant but meaningless. And problematic because authors wishing to confirm their biases and promote their hypotheses conveniently forget the warning that correlation is not evidence for causation and go on to promote their relationships as proof of effects. Often they seem to be successful in convincing regulators and policymakers that their serious relationships should result in regulations. Then there are the activists who don’t need convincing but will promote willingly and tiresomely these studies if they confirm their agendas.

Even random data can provide statistically significant relationships

The graphs below show the fallacy of relying only on statistically significant relationships as proof of an effect. The show linear regression result for a number of data sets. One data set is taken from a published paper – the rest use random data provided by Jim Jones in his book Regression Analysis: An Intuitive Guide.

All these regressions look “respectable.” They have low p values (less than the conventional 0.05 limit) and the R-squared values indicated they “explain” a large fraction of the data – up to 49%. But the regressions are completely meaningless for at least 7 of the 8 data sets because the data were randomly generated and have no relevance to real physical measurements.

This should be a warning that correlations reported in scientific papers may be quite meaningless.

Can you guess which of the graphs is based on real data? It is actually the graph E – published by members of a North American group currently publishing data which they claim shows community water fluoridation reduces child IQ. This was from one of their first papers where they claimed childhood ADHD was linked to fluoridation (see Malin, A. J., & Till, C. 2015. Exposure to fluoridated water and attention deficit hyperactivity disorder prevalence among children and adolescents in the United States: an ecological association).

The group used this paper to obtain funding for subsequent research. They obviously promoted this paper as showing real effects – and so have the anti-fluoride activists around the world, including the Fluoride Action Network (FAN) and its director Paul Connett.

But the claims made for this paper, and its promotion, are scientifically flawed:

1. Correlation does not mean causation. Such relationships in larger datasets often occur by chance – hell they even occur with random data as the figure above shows.
2. Yes, the authors argue there is a biologically plausible mechanism to “explain” their association. But that is merely cherry-picking to confirm a bias and there are other biologically plausible mechanisms they did not consider which would say there should not be an effect. The unfortunate problem with these sorts of arguments is that they are used to justify their findings as “proof” of an effect. To violate the warning that correlation is not causation.
3. There is the problem of correcting for cofounders or other risk-modifying factors. While acknowledging the need for future studies considering other confounders, the authors considered their choice of socio-economic factors was sufficient and their peer reviewers limited their suggestion of other confounders to lead. However, when geographic factors were included in a later analysis of the data the reported relationship disappeared. 

Confounders often not properly considered

Smith & Ebrahim (2002) discuss this problem an article  – Data dredging, bias, or confounding. They can all get you into the BMJ and the Friday papers. The title itself indicates how the poor use of statistics and unwarranted promotion of statical analyses can be used to advance scientific careers and promote bad science in the public media.

These authors say:

“it is seldom recognised how poorly the standard statistical techniques “control” for confounding, given the limited range of confounders measured in many studies and the inevitable substantial degree of measurement error in assessing the potential confounders.”

This could be a problem even for studies where a range of confounders are included in the analyses. But Malin & Till (2015) considered the barest minimum of confounders and didn’t include ones which would be considered important to ADHD prevalence. In particular, they ignored geographic factors and these were shown to be important in another study using the same dataset. Huber et al (2015) reported a statistically significant relationship of ADHD prevalence with elevation. These relationships are shown in this figure

Of course, this is merely another statistically significant relationship – not proof of a real effect and no more justified than the one reported by Malin and Till (2015). But it does show an important confounder that Malin & Till should have included in their statistical analysis.

I did my own statistical analysis using the data set of Malin & Till (2015) and Huber et al (2015) and showed (Perrott 2018) that inclusion of geographic factors showed there was no statistically significant relationship of ADHD prevalence with fluoridation as suggest by Malin & Till (2015). Their study was flawed and it should never have been used to justify funding for future research on the effect of fluoridation. Nor should it have been used by activists promoting an anti-fluoridation agenda.

But, then again, derivation of a statistically significant relationship by Malin & Till (2o15) did get them published in the journal Environmental Health which, incidentally, has sympathetic reviewers (see Some fluoride-IQ researchers seem to be taking in each other’s laundry) and an anti-fluoridation Chief Editor – Phillipe Grandjean (see Special pleading by Philippe Grandjean on fluoride). It also enabled the promotion of their research via institutional press releases, newspaper article and the continual activity of anti-fluoridation activists. Perhaps some would argue this was a good career move!

Conclusion

OK, the faults of the Malin & Till (2015) study have been revealed – even though Perrott (2018) is studiously ignored by the anti-fluoride North American group which has continued to publish similar statistically significant relationships of measures of fluoride uptake and measures of ADH or IQ.

But there are many published papers – peer-reviewed papers – which suffer from the same faults and get similar levels of promotion. They are rarely subject to proper post-publication peer-review or scientific critique. But their authors get career advancement and scientific recognition out of their publication. And the relationships are promoted as evidence for real effects in the public media.

No wonder members of the public are so often confused by the contradictory reporting, the health scares of the week, they are exposed to.

No wonder many people feel they can’t trust science.

Similar articles

Experts complain to funding body about quality of fluoride-IQ research

Science should never be protected from critical and rational discussion. Funding bodies should also be aware of problems in the research they fund. Image credit: The value of experience in criticizing research.

The scientific community was generally critical of the recent Canadian maternal neonatal fluoride – child IQ research (see expert reaction to study looking at maternal exposure to fluoride and IQ in children). But this has now taken a more serious turn.  Thirty academics and professional experts from health and dental institutions in the US, Canada, UK, Ireland, and Australia have formally complained to the US National Insitute of Environmental Health Science (NEHS) about the study.

This is highly important as the NEHS is the funding body for this research. If it takes seriously the criticisms of poor quality of the research and its bias it could well mean these study authors lose their funding.

I have covered professional criticism of this study in previous articles (and included some of my own critical comments). See:

• If at first you don’t succeed . . . statistical manipulation might help
• Politics of science – making a silk purse out of a sow’s ear
• More expert comments on the Canadian fluoride-IQ paper
• An evidence-based discussion of the Canadian fluoride/IQ study
• Fluoridation – A new fight against scientific misinformation
• Biostatistical problems with the Canadian fluoride/IQ study

Here is the letter to the NIEHS – readers can download and read it for themselves. I urge you to do this as there may well be a lot of misrepresentation circulating in the near future if anti-fluoride activists launch a campaign to discredit it.

Release of data and methodology requested

The letter requests the NIEHS:

“formally ask the Green authors to release the HIPAA-compliant, Research Identifiable File (RIF) data sets from their study, as well as a complete explanation of their methods and the computer program/codes used in their data management and analysis.”

This request is motivated by the fact that several of the study authors “have declined to respond affirmatively to requests from other researchers for access to the data and analytical methods they used.”

I know that study authors have gone even further – for example, asking that a university department pressure one of their research students to remove social media discussion of the study. Unfortunately, the student did remove his posts – but I can understand the power of institutional pressure.

I think such to such limiting of critical post-publication discussion is ethically unscientific as it inhibits true peer review. It’s made worse in this situation as the journal has a policy of restricting publishing any critiques of papers to four weeks after publication. The journal editor did refer to “the implications of this study” being “debated in the public arena” – but it appears that the authors are not exactly keen on that either.

Large range of problems with the Canadian study

The letter lists a number of problems with the Canadian study. These include:

1. Focusing on a subgroup analysis amid “noisy data”:
2. Modeling and variable anomalies:
3. Lacking data on relevant factors:
4. Omitting crucial findings:
5. Using invalid measures to determine individual exposures:
6. Defining the final study group:
7. Assessing the impact of fluoride exposure:
8. Reporting anomalies:
9. Internal inconsistency of outcomes:
10. Overlooking research that conflicts with the authors’ conclusions:

I urge readers who are interested in either of these aspects to refer to the letter for details of the problems. The letter includes a list of 30 references relevant to these problems and to criticisms of the study by other professionals.

Scientific politics

In Politics of science – making a silk purse out of a sow’s ear I raised the problems presented by scientific politics where poor studies are often promoted by journals, institutions, and authors. Maybe that is to be expected – science is a human activity and therefore subject to human problems like ambition and self-promotion.

Billboards like this misrepresent the Canadian research. But self-promotion and ambition of researchers and authors provide “authoritative” statements that activists use for such fake advertising.

However, in this case, scientific ambition and self-promotion have led to apparently “authoritative” statements by professionals that have been used to feed the scaremongering of anti-fluoride activists. These professionals may argue they are careful to qualify their statements but in the end, they must bear a lot of responsibility for the sort of completely misleading and false advertising activists have been promoting. Advertising which has serious consequences because of its scaremongering.

Scaremongering and scientific integrity

The letter also raises the problem of scaremongering in its final paragraph:

“. . . the Green article could generate unjustified fear that undermines evidence-based clinical and public health practices. So much is at stake. Hundreds of millions of people around the globe—from Brazil to Australia—live in homes that receive fluoridated drinking water. Hundreds of millions of people use toothpaste or other products with fluoride. Many millions of children receive topical fluoride treatments in clinical or other settings. Tooth decay remains one of the most common chronic diseases for children and teens, and fluoride is a crucial weapon against this disease. Decay prevalence could increase if a journal article unnecessarily frightens people to avoid water, toothpaste or other products fortified with fluoride.”

This letter by 30 high ranking professionals is extremely important. The concerns it raises are very relevant to scientific integrity and hence scientific credibility. I hope that the NIEHS and similar bodies will take on board the responsibility they have to ensure the work they fund is credible, expert, scientifically authentic and as free as possible from personal scientific ambitions and biases.

Similar articles

 

Fake weight-loss study example of wider problem

Click on image to enlarge

Another interesting article in the Conversation – Trolling our confirmation bias: one bite and we’re easily sucked in by Will Grant. It underlines a point  I have often made – that the sensible reader must approach the scientific literature intelligently and critically.
Grant describes a “scientific” prank which fooled many news outlets who reported the “scientific finding”, and, therefore, many readers.

“Last week science journalist John Bohannon revealed that the whole study was an elaborate prank, a piece of terrible science he and documentary film makers Peter Onneken and Diana Löbl – with general practitioner Gunter Frank and financial analyst Alex Droste-Haars – had set up to reveal the corruption at the heart of the “diet research-media complex”.”

The first trick

This was more than just planting a fictitious “science” story:

“To begin the study they recruited a tiny sample of 15 people willing to go on a diet for three weeks. They divided the sample into three groups: one followed a low carbohydrate diet; another followed that diet but also got a 42 gram bar of chocolate every day; and finally the control group were asked to make no changes to their regular diet.

Throughout the experiment the researchers measured the participants in 18 different ways, including their weight, cholesterol, sodium, blood protein levels, their sleep quality and their general well being.”

So – that was the first trick. “Measuring such a tiny sample in so many ways means you’re almost bound to find something vaguely reportable.” As Bohannon explained:

“Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out — the headline could have been that chocolate improves sleep or lowers blood pressure — but we knew our chances of getting at least one “statistically significant” result were pretty good.”

Publication

Now to get credibility they needed to publish in a scientific journal:

“But again, Bohannon chose the path that led away from truth, picking a journal from his extensive list of open access academic journals (more on this below). Although the journal, (International Archives of Medicine), looks somewhat like a real academic journal, there was no peer review. It was accepted within 24 hours, and published two weeks later.”

Now for the publicity

Bohannon then whipped up a press release to bait the media :

“The key, Bohannon stated, was to “exploit journalists’ incredible laziness” – to write the press release so that reporters had the story laid out on a plate for them, as it were. As he later wrote, he “felt a queazy mixture of pride and disgust as our lure zinged out into the world”. And a great many swallowed it whole.

Headlines around the world screamed Has the world gone coco? Eating chocolate can help you LOSE weight, Need a ‘sweeter’ way to lose weight? Eat chocolates! and, perhaps more boringly, Study: Chocolate helps weight loss.”

We should be concerned at the way the news media and reporters handle such matters:

“None did the due diligence — such as looking at the journal, looking for details about the number of study participants, or even looking for the institute Bohannon claimed to work for (which exists only as a website) — that was necessary to find out if the study was legitimate.”

This criticism, unfortunately, applies to almost anything in our news media. it really is a matter of “reader beware.”

Grant summarises the process that leads to such devious “science’ stories in the media:

• we’ve got researchers around the world who have taken to heart the dictum that the quantity of research outputs is more important than the quality
• we’ve got journal publishers at the high quality end that care about media impact more than facts
• we’ve got journal publishers at the no-quality end who exploit the desperation of researchers by offering the semblance of publication for a modest sum
• we’ve got media outlets pushing their journalists ever harder to fill our eyeballs with clickbaity and sharebaity content, regardless of truth
• and we’ve got us: simple creatures prone to click, read and share the things that appeal to our already existing biases and baser selves.

 Problem wider than the diet industry

Bohannon gives his prank as an example of a “diet research-media complex . . that’s almost rotten to the core.” I agree readers should be far more sceptical of such diet-related science stories. But the problem is far wider than that industry. I think is particularly relevant to any area where people are ideologically motivated, or their feelings of inadequacy or danger, can be manipulated.

Take, for example, the anti-fluoride movement. I have given many examples on this blog of science being misrepresented, or poor quality science being published and promoted by this movement. There are examples of anti-fluoride scientists doing poor quality research – often relying on “statistical fairy tales. Examples of using shonky journals to get poor quality work published. But also examples of such work making its way through inadequate journal peer-review processes.

These anti-fluoride researchers, and their allied activist groups, commonly use press releases to promote their shonky findings.  Social media like Facebook and Twitter are roped in to spread the message even more widely.

There is also a link with big business interests – in this case an active anti-fluoride “natural” health business-research-media complex.

So readers beware – there are people, businesses and ideological interests out there attempting to fool you. And they are not averse to using shonky or false science, biased press releases and lazy journalists to do this.

 See also: A rough guide to spotting bad science from Compound Interest (Click to enlarge).

Similar articles

 

 

Spotting Bad Science

Compound Interest has produced another great infographic in their series. This one helps us to spot bad science.

 Click to enlarge.  You can download the current version as a PDF here.

It is worth thinking about each of the suggested 12 criteria.

I particularly liked that it advises one to carefully evaluate scientific papers even when they are published in a reputable journal. A good journal and peer review is not a guarantee that the paper is faultless or that its findings can be accepted without proper consideration.

Dishonesty of intelligent design “research”

In my recent post Creationists prefer numerology to real scientific research I discussed the “research” approach used by those few scientists who are proponents of intelligent design. And I concluded:

“they ignore the normal honest research approach. They never advance a structured hypothesis, one that is consistent with intelligent design. They therefore never submit such hypothesis to any testing or validation.”

Michael Behe is Professor of Biological Sciences at Lehigh University in Pennsylvania. He works as a senior fellow with the Discovery Institute’s Center for Science & Culture.

Recently I noticed another blatant example of this lack of scientific honesty – the refusal to propose and test their own hypotheses of intelligent design. It’s a quote that seems to be going around the religious apologist bogs at the moment. For example, have a look at True Paradigm: Monday quote, The Big Bad Wolf, Theism and the Foundations of Intelligent Design – Page 13, or Still Speculating After All These Years at Contra Celsum.

It’s a quote from Michael J. Behe‘s book Darwin’s Black Box: The Biochemical Challenge to Evolution – this is the short form.

“The overwhelming appearance of design strongly affects the burden of proof: in the presence of manifest design, the onus of proof is on the one who denies the plain evidence of his eyes.”

Michael J. Behe, Darwin’s Black Box: The Biochemical Challenge to Evolution p 265.

Notice the problem?

Behe is asserting that he has no need to produce any evidence, outline a structured hypothesis, or do anything to test or validate his claim.

He simply has to make an assertion – based on nothing more than his claim of an “overwhelming appearance” (to him). Then it is up to those with different hypothesis to do all the work. To test his assertion (please note – a vague assertion – not a structured hypothesis) and prove him wrong.

Or else he declares his assertion correct by default!

Similar articles