Sfrgttk

A colleague has recently made the claim that a large p-value was not more support for the null hypothesis than a low one. Of course, this is also what I learned (uniform distribution under the null hypothesis, we can only reject the null hypothesis...). But when I simulate two random normal distributions (100 samples in each group) in R, my p-value is correlated to the difference (averaged over 30 repetitions) between the two means (with for example a T test or a Mann & Whitney test).

Why is my p-value, above the threshold of 0.05, correlated to the difference between the means of my two groups?

With 1000 repetitions for each x (difference between means/2) value.

My R code in case this is just a silly mistake.

pvaluetot<-NULL

xtot<-NULL

seqx<-seq(0,5,0.01)

for (x in seqx){

  ptemp<-NULL

  pmean<-NULL

  a<-0



  repeat{

    a<-a+1

    pop1<-rnorm(100,0+x,2)

    pop2<-rnorm(100,0-x,2)

    pvalue<-t.test(pop1,pop2)$p.value



    ptemp<-c(ptemp,pvalue)

    #print(ptemp)

    if (a==30)

      break

    }



  pmean<-mean(ptemp)

  pvaluetot<-c(pvaluetot,pmean)

  xtot<-c(xtot,x)

  print(x)

}



pvaluetot

xtot

plot(pvaluetot,xtot)

Why would you expect anything else? You don't need a simulation to know this is going to happen. Look at the formula for the t-statistic:
$t = frac{bar{x_1} - bar{x_2} }{sqrt{ frac{s^2_1}{n_1} + frac{s^2_2}{n_2} }}$

Obviously if you increase the true difference of means you expect $bar{x_1} - bar{x_2}$ will be larger. You are holding the variance and sample size constant, so the t-statistic must be larger and thus the p-value smaller.

I think you are confusing a philosophical rule about hypothesis testing with a mathematical fact. If the null hypothesis is true, you would expect a higher p-value. This has to be true in order for hypothesis testing to make any sense.

As you said, the p-value is uniformly distributed under the null hypothesis. That is, if the null hypothesis is really true, then upon repeated experiments we expect to find a fully random, flat distribution of p-values between [0, 1]. Consequently, a frequentist p-value says nothing about how likely the null hypothesis is to be true, since any p-value is equally probable under the null.

What you're looking at is the distribution of p-values under an alternative hypothesis. Depending on the formulation of this hypothesis, the resulting p-values can have any non-Uniform, positively skewed distribution between [0, 1]. But this doesn't tell you anything about the probability of the null. The reason is that the p-value expresses the probability of the evidence under the null hypothesis, i.e. $p(D|H_0)$, whereas you want to know $p(H_0|D)$. These two are related by Bayes' rule:
$$
p(H_0|D) = frac{p(D|H_0)p(H_0)}{p(D|H_0)p(H_0)+p(D|neg H_0)p(neg H_0)}
$$
This means that in order to calculate the probability you're interested in, you need to know and take into account the prior probability of the null being true ($p(H_0)$), the prior probability of the null being false ($p(neg H_0)$) and the probability of the data given that the null is false ($p(D|neg H_0)$). This is the purview of Bayesian, rather than frequentist statistics.

As for the correlation you observed: as I said above the p-values will be positively skewed under the alternative hypothesis. How skewed depends what that alternative hypothesis is. In the case of a two-sample t-test, the more you increase the difference between your population means, the more skewed the p-values will become. This reflects the fact that you're making your samples increasingly more different from what is plausible under the null, and so by definition the resulting p-values (reflecting the probability of the data under the null) must decrease.