When a population parameter is estimated through a sample statistic (e.g., the sample mean as an estimate of the population mean), we obtain different results with different samples.
The different values for the sample mean obtained with different samples constitute also a random variable. As such it has a mean or expected value and a variance.
The distribution of the statistic (in this example the sample mean) is called the sampling distribution of the statistic.
Additional information can be found in David Lane's text.
There is a very important result in statistics that states:
"The sampling distribution of the sample mean can be approximated a a normal probability distribution when the sample size is large, regardless of the distribution of the original random variable."
This says, that even when a variable follows a uniform distribution, or an exponential distribution, the probability distribution of the sample mean computed from samples coming from those distribution will be normal as the sample size increases. This usually happens as the size of the sample is greater than 30.
The importance of this theorem is found at the time of making inferences about the value of the sample mean. It means that for inference purposes we can use the normal distribution if the sample size is large regardless of the distribution of the original variable. If the original variable was normally distributed, then the central limit theorem does not need to be applied.
HyperStat Online. Copyright © 1993-2000 David M. Lane.