Problem
Descriptive Statistics
There were 219 papers, of which 180 papers contained codable samples (EP 76; E&HB 104). Thirty-nine pa- pers could not be coded because they consisted of, for instance, mathematical models, work on non-humans, or literature reviews. Within the 180 codable papers, there were 311 samples, and the median number of samples per paper was 1. The mean sample size was 4094 but this was driven by one extremely large sample (N = 927,134). The median sample size was 186 but sample sizes varied substantially (minimum 11; first quartile 96.5; third quartile 334.5; maximum 927,134).
Figure 1 shows the distribution of samples by geo- graphical region. The majority of samples were from North America (153), followed by Europe (93) and Asia (37). Of the Asian samples, the majority were from Japan (11), followed by China (7) and Israel (6). There were only 6 samples from Africa (4 from Tanzania, 1 from Namibia, 1 from Nigeria) and only 8 from Latin America and the Caribbean (2 from Guatemala; 2 from Curaçao; 2 from Bolivia; 1 commu- nity from Northern-Brazil/Southern Guyana/East Venezuela and 1 undefined [Latin American students studying in Germany]). There were 7 samples from Australia and 1 from Oceania (excluding Australia): Fiji Island. Only 6 samples were Cross-Cultural (con- taining samples from more than five different countries). Combining the figures, we found that around 8 out of 10 samples were from WEIRD populations (81%, Europe/North America/Australia), and that 87% of the samples used were from developed regions (following the UN classification; United Nations 2013).
In terms of sample type, 113 of the 311 samples were Western student samples, while 24 were non- Western student samples; 60 samples were online paid crowdsourced, while 20 were unpaid crowdsourced. Thus, 70% of the samples were either online samples or student samples. Twenty-five samples were based on children (21 from Western and 4 from non-Western pop- ulations). Only a small fraction of the samples consisted of non-Western adults who were not students (24 out of 311 samples, or 8%).
Are Samples from Certain Geographical Locations Larger than Others?
Given that there was only one sample from Oceania (exclud- ing Australia), we combined this with Australia for the analysis of variation in sample sizes between regions (see ESM for additional analyses using this combination). Variation in sample size between geographical regions was not statistically significant (Kruskal-Wallis test: χ2(6) = 10.095, p = .12). Following adjustment for multiple testing, the median sample size was found to be significantly larger for cross-cultural samples (which, according to our cod- ing criteria, had to contain data from more than five different countries) than for Latin American and Caribbean samples (p = .037). The ESM contains all post-hoc multiple compari- sons (all remaining p values > .09; see ESM).
Are some Types of Samples Larger than Others?
The sample sizes differed significantly according to type. Post-hoc comparisons adjusted for multiple testing using the Benjamini- Hochberg procedure showed that online (paid crowdsourced) and online (unpaid crowdsourced) samples tended to be larger than other types of samples.
Answer the following question:
I. Consider one sample analyzed in Pollet & Saxton. Identify 3 reference classes this group belongs to. Are these reference classes homogeneous?
II. Identify 1 reference class which you share with the sample and 1 reference class which you don't share with the sample. Do you think results derived from these studies apply to you? Why/why not? Put your worry in either frequentist or Bayesian terms (just one of them).