Consider an experiment with success probability p. The entropy is
If p is unknown a natural (and best) estimate after n repetitions is the relative frequency of successes, p∗n = Xn/n, where Xn equals the number of successes. In order to find an estimate of H(p) it seems reasonable to try H(p∗n). Determine the asymptotic distribution of H(p∗n) as n → ∞.
Do not forget to distinguish between the cases p ≠ 1/2 and p ≠ 1/2.
We have thus replaced the estimate of the function with the function of the estimate.