Professor Duneier and his assistants have thoroughly graded thousands of finals and midterms, comparing their scores with the peer graders'. When he sees the first batch of midterms, he realizes that some students had given unexpected responses that would not have earned various points on his planned rubric, despite their clearly understanding the material. So he tweaked the rubric, allowing for extra "makeup" points on some problems. But the computer tallied the regular and makeup points together, provide some students more total points than the exam was worth.
"I had to announce to the students that some had gotten scores that were higher than they should have been," he said. "And as data, the midterm scores are useless. But it helped us learn more about writing rubrics."
Testing, Testing
Now, months after the course ended, he and his assistants are hand-scoring the final exams, checking the scores they assign (he avoids the word "grades") against those given by students. So far, he has found an impressive correlation of 0.88. The average peer score was 16.94 of 24 possible points, compared with an average teaching-staff score of 15.64. Peer graders give more accurate scores on good exams than bad ones, they found, and the lower the score, the more variance among graders.
As with other MOOCs, less than five percent of those who enrolled in the sociology course completed it: 2,200 midterm exams and 1,283 final exams were submitted. Some students listened to all the lectures and did all the readings but did not take exams. There was no practical reason to take the exams, since Princeton - unlike Udacity, edX or other universities working with Coursera - does not provide certificates of completion.