Information Theory Assignment
The concept of entropy can be applied to any language by assuming that the alphabetical characters are generated randomly by a data source. To better understand this concept, you need to do the following for both English and Arabic languages:
Part 1:
1. Copy form the internet an article of your choice, the article should be at least 1000 words, paste the article in a text file.
2. Convert all capital letters (for the English) to small letters and remove all punctuation marks from the article. Just leave the spaces.
3. Write a Matlab code that generates the probability distribution function (PDF) of your article.
4. Write a Matlab code that computes the entropy of the text.
5. Compare the results of both languages with each other.
Part 2:
During your studies of the Information Theory, you have seen that the entropy of a binary source follows the horse-shoe function given that we know the probability (p, 1 - p) of each event accurately. In this part you are required to:
1. Generate the horse-shoe function using Matlab assuming that p is known perfectly.
2. Generate the horse-shoe function using Matlab by generating N samples of a binary random variable. The function should be generated using four different values of N (10, 100. 1,000, 10,000).
3. Compare the results for the four cases (N = [10, 100. 1,000, 10,000]) with the ideal one by plotting them on the same figure.
4. Compute and draw the KL distance between the ideal case (p) and the other four cases where p is obtained using simulation. What is your conclusion? Show your solution in details.