Question: Suppose we take a look at the more than 10,000 mutual funds on sale to the investing public. Any Internet investment site worth clicking on will tell you which funds produced the highest returns over the past (say) three years. In 2005, one site claimed that the Bruce Fund was among the top 1% over the three preceding years. If we had bought this fund in 2002, we would have gained 42.4% per year. Comparing this return with the average over this period for all funds, we find that it is significantly higher. Doesn't statistical significance suggest that the Bruce Fund is a good investment?
In this chapter we will take a careful look at what statistical confidence and statistical significance do and do not mean. We will discuss some abuses of statistical inference. By the end of this chapter you will be able to answer the question of whether there was strong evidence that the Bruce Fund was a sound investment in 2005.