> So the researcher tests if omega 3 effects cardiovascular outcome in these hundred people by adding a lot more fish oil to the diet of these 100 people. Since only one of them really needs it, the P value will be insignificant and everyone will say fish oil does nothing. Yet for that one person it was literally everything.
But... that's not a problem with the use of the p-value, because that's (quite probably) a correct conclusion about the target (unrestricted) population addressed by the study as a whole.
That's a problem with not publishing complete observations, or not reading beyond headline conclusions to come up with future research avenues. That effects which are not significant in a broad population may be significant in a narrow subset (and vice versa) are well-known truths (they are the opposites of the fallacies of division and composition, respectively.)
But... that's not a problem with the use of the p-value, because that's (quite probably) a correct conclusion about the target (unrestricted) population addressed by the study as a whole.
That's a problem with not publishing complete observations, or not reading beyond headline conclusions to come up with future research avenues. That effects which are not significant in a broad population may be significant in a narrow subset (and vice versa) are well-known truths (they are the opposites of the fallacies of division and composition, respectively.)