One thing that has struck me listening to talks at the European Congress of Epidemiology is the incredible weight given to the phrase “statistically significant”. This is an old chestnut among theoreticians in the area, so my surprise perhaps indicates more about my selective contact with epidemiology to date than anything else. It is nonetheless interesting to see the work this strange concept does.

The most striking example was in an interesting talk on risk factors for colorectal cancers. A slide was displayed showing results of a case control study. For every one of the 8 or so risk factors, incidence among cases was higher than controls. However, the speaker pointed out that only some of these differences were statistically significant.

This struck me as very strange. The level of statistical significance is more or less arbitrary – perhaps not entirely, but arbitrary in the same way as specifying a certain height for “short”. In this context, that means that the choice of risk factors to ignore was also, in the same sense, arbitrary. Moreover, the fact that the difference was the same way in all the risk factors (ie higher exposure in cases than controls) also seemed, to my untutored eye, to be the sort of unlikely coincidence one might wish to investigate further.

In a way, that is exactly what came next. One of the “insignificant” factors turned out – and I confess I did not follow how – to interact significantly with another (the two being fibre and calcium intake).

I am not sure that any of this is problematic, but it is certainly puzzling. The pattern is not unique to this talk. I have seen more than one table presented of variables potentially associated with an outcome, with the non significant ones then being excluded. On many occasions this must surely be a good, quick way to proceed. It seems like a strange exercise, to my untutored eye, if some non significant differences are studied further anyway. But that is surely an artefact of my lack of understanding.

I am less sure that my lack of understanding is to blame for other doubts, however. Where a number of risk factors are aligned, it seems arbitrary to ignore the ones that fail a certain level of statistical significance. The fact of alignment is itself some evidence of a non chance phenomenon of some kind. And, of course, the alignment might indicate something important, for example an as yet unthought of causal factor. The non significant factors could be as useful as the significant ones in detecting such a factor, by providing further means of triangulation.