Introduction -
In this assignment, you will download annual reports for Apple, Inc. from EDGAR and analyse sentiment. In Task 1 you will download and structure the annual reports. In the remaining tasks you will analyse sentiment measures. The main goal is to critically understand what parameters and choices influence the level of computed sentiment.
Except Task 1, all tasks use the data gathered. In case you are unable to complete the initial data gathering and cleaning tasks, you can use the data we provide, apple_10k.Rdata. The file contains a vector with each element being an individual annual report without having applied any cleaning task. The names of the vector indicate the submission date. In this assignment, there are several steps that can yield different results depending on the actual coding. Hence, as long as you fullfill all steps, it does not matter if your downloaded data is not exactly the same as the one provided by us.
Task 1: Download annual reports
Download all annual reports for Apple, Inc. from 1994 to 2016 using EDGAR. Save the raw teat file without any cleaning.
Note: If you download an annual report (10-K) from EDGAR, the obtained text file includes several additional filings other than the annual report. Make sure that 3ou only use the annual report for later analysis. Hint: use the structure of the downloaded file to identify the annual report.
Task 2: Sentiment and document cleaning
In this task, you evaluate the impact of document cleaning tasks on sentiment computation. We only focus on the Harvard-IV and Loughran-MacDonald dictionary. Using the SentimentAnalysis package, this refers to sentimentGI and sentimentLM. Please use all documents in the whole corpus for this task.
Compute sentiment for each individual annual report using...
- the raw text files.
- the text files after removing html tags.
- the text files after removing html tags, excess white space, numbers, punctuation, removing words with less than 4 and more than 30 characters, and removing stopwords.
Make two time series plots:
- Plot a time-series of the three sentimentGI measures computed in the previous step.
- Plot a time-series of the three sentimentLM measures computed in the previous step.
Why does the cleaning measures impact the level of sentiment?
What explains the one visible drop in one sentence measure?
Task 3: Terms defining sentiment
In this task, you evaluate the terms defining the individual sentiment measures. Please use all documents in the whole corpus for this task with html tags removed and cleaning steps of your choice. Answer following questions:
- What are the 25 most common positive terms based on the Harvard-IV and Loughran-MacDonald dictionary using the whole corpus?
- How many of them are present in the other dictionary? (Harvard-IV versus Loughran-MacDonald)
- Interpret your findings.
Task 4: Comparison between different sentiment methods - advanced
In this task, you will compare two different measures of computing sentiment. In particular, you will evaluate if the correlation between both measures is different for sentences with high or low sentiment. We only focus on the Harvard-IV and Loughran-MacDonald dictionary. Using the SentimentAnalysis package, this refers to sentimentGI and sentimentIA Furthermore, we are only considering the most recent annual report by Apple, Inc.
Split the annual report in sentences. Exclude sentences of unreasonably short and long length. Compute sentiment for each sentence. Compute following tasks:
Normalize each sentiment measures accross documents by deducting the mean and deviding by the standard deviation. (x - mean(r))/std(x). After this step, each sentiment measure has mean zero and standard deviation one and hence they are comparable in magnitude and variation. Use the standardised version for later computations.
Sentiment and sentence length:
- Split all sentences in five bins based on sentence length. Set the breakpoints at the quintiles so that each bin includes a fifth of the data.
- For each bin, compute the average sentiment124 and sentimentGI, and visualize.
- For each bin, compute the correlation between sentimentCI and sentlmentLM. Visualize with a plot.
Sentiment and its level:
- Split the sentences in six bins based on sentlmentGI falling in one of the following intervals: (- inf : -0.51/(-0.5 : 01/(0 : 0.51/(0.5 : 1.51/(1.5 : inf). How many observations are in each individual bin?
- For each bin, compute the average sentimentLM, and visualize.
- For each bin, compute the correlation between sentimentGI and sentlmentLM. Visualize with a plot.
Interpret your findings.
Assignment Files -
https://www.dropbox.com/s/zbvl4uulp6nudtl/Assignment%20Files%20-%20Task%203%20and%20Task%204.rar?dl=0