Discuss the below:
Scenario
Statistical significance is found in a study, but the effect in reality is very small (i.e., there was a very minor difference in attitude between men and women). Were the results meaningful?
An independent samples t test was conducted to determine whether differences exist between men and women on cultural competency scores. The samples consisted of 663 women and 650 men taken from a convenience sample of public, private, and non-profit organizations. Each participant was administered an instrument that measured his or her current levels of cultural competency. The cultural competency score ranges from 0 to 10, with higher scores indicating higher levels of cultural competency. The descriptive statistics indicate women have higher levels of cultural competency ( M = 9.2, SD = 3.2) than men ( M = 8.9, SD = 2.1). The results were significant t (1311) = 2.0, p <.05, indicating that women are more culturally competent than are men. These results tell us that gender-specific interventions targeted toward men may assist in bolstering cultural competency.Scenario
The p-value was slightly above conventional threshold, but was described as "rapidly approaching significance" (i.e., p =.06).
An independent samples t test was used to determine whether student satisfaction levels in a quantitative reasoning course differed between the traditional classroom and on-line environments. The samples consisted of students in four face-to-face classes at a traditional state university (n = 65) and four online classes offered at the same university (n = 69). Students reported their level of satisfaction on a fivepoint scale, with higher values indicating higher levels of satisfaction. Since the study was exploratory in nature, levels of significance were relaxed to the .10 level. The test was significant t(132) = 1.8, p = .074, wherein students in the face-to-face class reported lower levels of satisfaction (M = 3.39, SD = 1.8) than did those in the online sections (M = 3.89, SD = 1.4). We therefore conclude that on average, students in online quantitative reasoning classes have higher levels of satisfaction. The results of this study are significant because they provide educators with evidence of what medium works better in producing quantitatively knowledgeable practitioners
Instructions: Critically evaluate the scenario you selected based upon the following points:
Critically evaluate the sample size.
Critically evaluate the statements for meaningfulness.
Critically evaluate the statements for statistical significance.
Based on your evaluation, provide an explanation of the implications for social change.
a) Sample size:
As said by Mugenda (1999), a sample size of five percent of the whole population is already enough for the analysis. In this case, the population of the school is not given. However, it is reasonable to presuppose that 132 represents more than or equal to 5% population. If the study is hypothesis-driven, the sample size (n=65 and n=69) in both control and experimental setups, respectively, was too big for the statistical test used that was the t-test. Since the sample size is larger than 30, Z-test may be used instead of t-test, if normal distribution is applied.
Statements for meaningfulness:
Instead of mentioning "rapidly approaching significance", it is more appropriate to say that the P-value of 0.06 is on the edge of significance. The former statement suggests that the P-value is reducing and to make it even worse-sounding, the word "rapidly" has been used, which is not actually statistically correct.
Statistical significance:
Using t-test must determine if there was a significant difference between the traditional classroom (control group) and on-line environments (experimental group). However, it was noted that the study is exploratory and hypothesis testing is not necessary. Hypotheses statements are not used since the research is not descriptive. With absence of hypotheses statements, p-values were not really meaningful and only served as a rough guide. P-values and their critical counterparts are used in descriptive and correlation forms of research. While it is all right to conduct exploratory analyses on sample plots positioned in accordance with a thorough, purposive sampling design, such meticulous placement is not compulsory. In exploratory research, the researcher is only interested in providing details where very little is known about the phenomenon. It may use various procedures such as trial studies, interviews, group discussions, experiments or other tactics for the purpose of gaining information. Therefore the statement, "Since the study was exploratory in nature, levels of significance were relaxed to the .10 level", is not statistically sound.
The statement, "The test was significant t (132) = 1.8, p = .074, wherein students in the face-to-face class reported lower levels of satisfaction (M = 3.39, SD = 1.8) than did those in the online sections (M = 3.89, SD = 1.4)", is not just statistically incorrect, but also confusing and misleading. T-test would not be applicable here as we have two groups which are further subdivided into 4 subgroups each. Samples are to be taken from each sub group and an Analysis of Variance (ANOVA) may be performed. Furthermore, traditional threshold of 0.05 is always recommended for use.
Implications for social change:
The first scenario showed great depth of confirmation of how p-values are generally misunderstood and misused. With more stress and importance on the subject matter, these errors can be avoided in future. Positive social changes will have a great impact in appreciating and understanding the true issue.
The result of the study is important in determining how the investigator can optimally explain or describe the variation in the data set through data-diving. The investigator may find patterns in the students' answers that may propose evidences to educators on what medium works more effectively in the classroom.
b) Sample Size:
Sampling, in this scenario, confirmed the results but seemed to be so small although it represented the public, private and non-profit sectors. In collecting quantitative data, a statistical formula must be used to give a rough estimate of the sample size that is needed. There are always errors in sampling that reflect the differences arising between statistics and the population, if the sample is not an accurate representative of the entire population from which it was taken from. By increasing the sample size, the error can be minimized. It can even be eliminated by interviewing the whole population.
It can therefore be agreed that even though representative of the particular sectors of the employees, a sample size of n=432 is quite small. The findings may remain valid and relevant as the level income is always perceived to increase job satisfaction. This is because a large sample is recommended if there is high variability and small sample if variability is minimal.
Statements for meaningfulness:
The findings indicated that as the level of income increased, the job satisfaction increased as well. The findings reflected a reality in practical terms. This can be the reason why people fight for promotions, as the higher you move in the organization hierarchy, the greater the income, resulting to greater satisfaction. Also, people compete for jobs in high paying companies than companies known to be low at remunerating. To a great deal, this is meaningful and reflects the societal inclination towards money.
Statistical significance:
The result of the test demonstrated a strong positive correlation between the two variables, r =.87, p < .01, showing that as level of income increases, job satisfaction increased as well. The findings of this result were thus statistically significant because it is confirmed by the strong correlation of .87 or 87% and further by the statistical significance of .01 almost closing to .00 which indicates perfect linear relationship. The null hypothesis is, thus, rejected.
This implies that it is very sure that the relationship is real, and thus, the conclusions that the levels of satisfaction among employees are determined by the level of income. There lies a greater validity as the levels of significance remained at .05 or 5%. Thus, the researcher is 95% confident that the increase in the level of income increases job satisfaction.
Implications for social change:
The findings indicated that as the level of income increased, the job satisfaction increased as well, and this may be against the socialist-economist people who have put forward a strong argument that other factors such as the nature of job, work environment, management and the hierarchical structures, apart from money, play a key role to job satisfaction. Most of the economists believe that favorable work environments, flat organizations, less bureaucratic and co-operative management are essential in determining employee satisfaction. Nevertheless, money is still of primary importance to most people, and it can be concurred that the level of income is the determinant of job satisfaction as per the findings .Thus, definitely, as life becomes harder, a means of living (money) is always what the employees are chasing.