Can peer grading actually work?

Longtime readers may remember an essay I did for Inside Higher Education last year called “Peer Grading Can’t Work.” It came about as a result of my experiences in Jeremy Adelman’s World History MOOC and I think it may be the second best thing I’ve ever written (after this, of course). Therefore, I was intrigued to learn that Stephanie McCurry’s History of the Slave South MOOC from the University of Pennsylvania/Coursera has tried to fix the flaws that their MOOC predecessors found in earlier peer grading systems.

Since I have no time for anybody’s MOOC these days, Ben Wiggins, Associate Director of Online Learning at the University of Pennsylvania’s School of Arts and Sciences and Lecturer in Penn’s Department of History and Department of History and Sociology of Science, graciously agreed to describe their peer grading system for MOOC-obsessed MOLB readers of all stripes. I have promised that there will be no rebuttals from me here. I simply asked Ben to come back later and tell us all how their system worked.

***

It is immensely difficult to create a massive open online course in the humanities and social sciences that approximates a traditional brick-and-mortar offering. This should come as no surprise given MOOCs’ origin (or to anyone reading Jonathan’s site). All three major MOOC platforms (Udacity, Coursera, and EdX) have their roots in computer science departments. And even their connectivist predecessors were created by academics in computing, too.

MOOCs emerged from the sciences because the sciences are scalable. They’re scalable on campus and they are scalable beyond campus. Sure, labs and more nuanced evaluation based on shown work are lost in multiple-choice examinations and task-based programming assignments, but the core forms of STEM assessments translate well to MOOCs. The humanities and social sciences, however, resist machine grading. Writing—synthetic expressions of the complexity of histories, societies, cultures, and creative works—rests at the center of evaluation in almost every single discipline and interdiscipline in the humanities and social sciences.

MOOC platforms are not ignorant of the centrality of writing in the humanities and social sciences, but their approximation of the campus experience—essentially relying on peer grading—has left much to be desired. I’ve never liked peer grading in my classroom and I think it’s even worse when also decoupled from instructor evaluation, as is the necessity with course enrollments regularly in the tens of thousands and instruction teams regularly less than half a dozen.

When grading is left to students, the results are mixed. Indeed, mixed may be a perfect term for peer assessments, since, as Jonathan has shown over at Inside Higher Ed, the quality of student grading varies widely. Efforts have already started to calibrate peer grading—Coursera has been a leader here—but this process takes a great deal of labor on the instruction team’s part and requires multiple offerings of the course before instruction team grades and student grades begin to correlate with much significance.

When I came to Online Learning in Penn’s School of Arts and Sciences early last summer, the first project that landed on my desk was to guide production on a new MOOC entitled History of the Slave South to be taught by history professor, Stephanie McCurry. I was excited by the assignment since my background is in the historical study of race, but I knew early on, too, that it was going to be a difficult course to construct. How would we keep the discussions respectful and on point? How could we teach historical skills in a class in which we could not assign any closed-access secondary sources? How could we assign essays without instructor or TA grades?

From the beginning of my work on History of the Slave South, that last question gnawed at me. How to construct an effective, enlightening peer review became one of two questions—the other being the translation of lab sessions to an online environment—I began to ask everyone I met with an interest in online learning. The months ticked away on the calendar and the January launch date approached. It was not until the Educause Conference in October that I found a promising lead.

There, Thomas Evans, Evonne Halasek, and Jennifer Michaels, all of The Ohio State University, presented on their experience with peer review in their MOOC on writing. This team had put a great deal of effort into constructing a peer review process that would benefit student learning, not simply replace instructor grading. And though they did not find a magical secret to perfecting peer assessment, they did find some promising directions to take the process in. The one that caught my attention was the success they found in description. Their peer review had students complete three tasks: description, assessment, and suggestion. In this describe-assess-suggest structure, they found high praise from students for the utility of descriptions their peers offered. It acted as a sort of mirror. It allowed students to see if they accomplished what they set out to accomplish. This focus on description immediately reminded me of a tactic I had used in my own on campus courses—the three questions of The Ignorant Schoolmaster.

Written by French philosopher Jacques Ranciere in 1987 (and first translated to English in 1991), The Ignorant Schoolmaster: Five Lessons in Intellectual Emancipation builds on Joseph Jacotot’s early-nineteenth-century panecastic pedagogy to argue for the “equality of intelligences.” Jacotot—the protagonist-muse of Ranciere’s text—was subdirector and professor at the École Polytechnique in Dijon and it was there where he crafted a pedagogy of “universal teaching,” which posited that each student is capable of teaching oneself as well as all others. We must free ourselves from the “masters” of traditional classrooms argues Jacotot and echoes Ranciere. Only then can we can achieve “intellectual emancipation.”

All a bit heady, no? And certainly a critique of expertise that does not sit well with historians like myself who reside in an institutional economy based so deeply on demonstrations of expertise.

For History of the Slave South, we did not leave behind the “master.” I cannot imagine a course like this without foundational lecture videos, highly curated texts, and refined discussion questions. Leaving students with no prerequisite knowledge of slavery’s history to teach each other is a scary proposition (though, a MOOC community might do better than “masters” in Texas). However, there is a place for Jacotot’s method in humanities and social sciences MOOCs—in peer review.

To teach his “students,” Jacotot asked them only three questions:

o What do you see here?
o What do you think about it?
o What do you make of it?

These questions, like the first stage of the Ohio State MOOC’s peer review, promote learning through description. This is an ideal method for a peer review in which the level of historical context about the subject at hand varies greatly from student to student. Our first assignment, for instance, tasks students with building tables from the Slave Voyages Database and interpreting them. We can’t assume whether or not a student knows if a peer’s interpretation is “correct.” Indeed, much of the point of the assignment is to expose the limits of interpretation in numerical representations of human lives. In essence, there is no correct answer. So rather than make our MOOC students into an army of pseudo-experts or TA proxies, we’ve tried to make their feedback more useful than judgmental. We’ve stopped calling it peer review or peer grading and, instead, termed it “peer reflection.” We want these reflections to act as a mirror to their peers. We want our writers to view their peers’ reflections as a way to see if they accomplished what they set out to accomplish. We significantly tempered the grading of our writing assignments so that it is now simply an “unsatisfactory,” “satisfactory,” “accomplished” scale. It is our hope—and here I must thank Stephanie and her teaching assistant, Roberto Saba, for fully committing to what I’m sure was a strange-sounding pitch with a radical nineteenth-century teaching method at its core—that a peer reflection system based in description will create a more useful experience for our learners.

After months of fretting and weeks of refining, Stephanie, Roberto, and I crafted the following instructions:

When assessing your peers’ work, please follow the instructions below:
1. Identify and describe your peer’s argument; what does it communicate to you?
2. Identify and describe your peer’s use of historical evidence; how does the evidence support the argument?
3. What makes your peer’s analysis persuasive? How could it be stronger?
To receive credit for your peer reflection, you must answer all three questions. Your reflection is required to be at least 150 words long and should not exceed 300.
In addition to this narrative feedback, please rate your peer’s essay as “unsatisfactory,” “satisfactory,” or “accomplished.”

We are under no delusions that our peer reflection will ever be as rich as most instructor feedback. But to claim humanities and social science MOOCs as anything even nearing an approximation of our campus offerings, we need to experiment with creative solutions in spaces in which massive, open courses have the greatest limitations. If we cannot find solutions to the conundrums that plague humanities and social sciences MOOCs soon, then the humanities and social sciences will once again falter where the STEM disciplines excel.

PS You can follow Ben on Twitter at @WigginsBenjamin.