We use 22,281 parameter estimates from 368 different fields of economics research to evaluate the statistical power and excess statistical significance of 31 major economics general interest and discipline journals . The median statistical power of the major economics journals is very low (only 7%) and the excess statistical significance is quite high (19%). When power is this low and importance is this high, it raises serious questions about the reliability of economic research. We find that 26% of all reported results underwent some selection process for statistical significance, and 56% of statistically significant results were selected as statistically significant. Selection bias was high in the top five journals, with 66% of statistically significant results selected as statistically significant. Much of the empirical evidence reported in major economic journals can be misleading. Results reported as statistically significant are more likely to be misleading (false positives), and results that are not statistically significant are much more likely to be misleading (false negatives). We also compared observational and experimental studies and found that the quality of experimental economic evidence was significantly higher.
It's from a new paper by Zohid Askarov, Anthony Doucouriagos, Hristos Doukouriagosand TD Stanley.
via my colleague Jonathan Schultz.