Poor quality in the reporting and use of statistical methods in public health – the case of unemployment and health

Fredrik Norström

It has previously been reported that many research articles fail to fulfill important criteria for statistical analyses, but, to date, these reports have not focused on public health problems. The aim of this study was to investigate the quality of reporting and use of statistical methods in articles analyzing the effect of unemployment on health. Forty-one articles were identified and evaluated in terms of how they addressed 12 specified criteria. For most of these criteria, the majority of articles were inadequate. These criteria were conformity with a linear gradient (100 % of the articles), validation of the statistical model (100 %), collinearity of independent variables (97 %), fitting procedure (93 %), goodness of fit test (78 %), selection of variables (68 % for the candidate model; 88 % for the final model), and interactions between independent variables (66 %). Fewer, but still alarmingly many articles, failed to fulfill the criteria coefficients presented in statistical models (48 %), coding of variables (34 %) and discussion of methodological concerns (24 %). There was a lack of explicit reporting of statistical significance/confidence intervals; 34 % of the articles only presented p-values as being above or below the significance level, and 42 % did not present confidence intervals. Events per variable was the only criterion met at an undoubtedly acceptable level (2.5 %). There were critical methodological shortcomings in the reviewed studies. It is difficult to obtain unbiased estimates, but there clearly needs to be some improvement in the quality of documentation on the use and performance of statistical methods. A suggestion here is that journals not only demand that articles fulfill the criteria within the STROBE statement, but that they include additional criteria to decrease the risk of incorrect conclusions being drawn.