to 2.0, our computed value of zM came out to be 1.0, which was not statistically significant. However, is it possible to increase the size of zM without changing the observed distance between means? Absolutely. Consider what happens to the size of zM as we change the magnitude of either σ or n, or both. First, we consider how zM is defined in part as a function of σ. For convenience, we assume a sample size still of n = 100. Consider now three hypothetical values for σ: 2, 10, and 20. Performing the relevant computations, observe what happens to the size of zM in the case where σ = 2:
The resulting value for zM is quite large at 10. Consider now what happens if we increase σ from 2 to 10:
Notice that the value of zM has decreased from 10 to 2. Consider now what happens if we increase σ even more to a value of 20 as we had originally:
When σ = 20, the value of zM is now equal to 1, which is no longer statistically significant at p < 0.05. Be sure to note that the distance between means
What this means is that given a constant distance between means
Suppose now we again assume the distance between means
With a sample size of 16, the computed value for zM is equal to 4. When we increase the sample size to 49, again, keeping the distance between means constant, as well as the population standard deviation constant, we obtain:
We see that the value of zM has increased from 4 to 6.9 as a result of the larger sample size. If we increase the sample size further, to 100, we get
and see that as a result of the even larger sample size, the value of zM has increased once again, this time to 10. Again, we need to emphasize that the observed increase in zM is occurring not as a result of changing values for
2.28.2 The Make‐Up of a p‐Value: A Brief Recap and Summary
The simplicity of these demonstrations is surpassed only by their profoundness. In our simple example of the one‐sample z‐test for a mean, we have demonstrated that the size of zM is a direct function of three elements: (1) distance
The important point here is that a large value of zM does not necessarily mean something of any practical or scientific significance occurred in the given study or experiment. This fact has been reiterated countless times by the best of methodologists, yet too often researchers fail to emphasize this extremely important truth when discussing findings:
A p‐value, no matter how small or large, does not necessarily equate to the success or failure of a given experiment or study.
Too often a statement of “p < 0.05” is recited to an audience with the implication that somehow this necessarily constitutes a “scientific finding” of sorts. This is entirely misleading, and the practice needs to be avoided. The solution, as we will soon discuss, is to pair the p‐value with a report of the effect size.
2.28.3 The Issue of Standardized Testing: Are Students in Your School Achieving More Than the National Average?
To demonstrate how adjusting the inputs to zM can have a direct impact on the obtained p‐value, consider the situation in which a school psychologist practitioner hypothesizes that as a result of an intensified program implementation in her school, she believes that her school's students, on average, will have a higher achievement mean compared to the national average of students in the same grade. Suppose that the national average on a given standardized performance test is equal to 100. If the school psychologist is correct that her students are, on average, more advanced performance‐wise than the national average, then her students should, on average, score higher than the national mark of 100. She decides to sample 100 students from her school and obtains a sample achievement mean of
