Can peer grading actually work?

17 02 2014

Longtime readers may remember an essay I did for Inside Higher Education last year called “Peer Grading Can’t Work.” It came about as a result of my experiences in Jeremy Adelman’s World History MOOC and I think it may be the second best thing I’ve ever written (after this, of course). Therefore, I was intrigued to learn that Stephanie McCurry’s History of the Slave South MOOC from the University of Pennsylvania/Coursera has tried to fix the flaws that their MOOC predecessors found in earlier peer grading systems.

Since I have no time for anybody’s MOOC these days, Ben Wiggins, Associate Director of Online Learning at the University of Pennsylvania’s School of Arts and Sciences and Lecturer in Penn’s Department of History and Department of History and Sociology of Science, graciously agreed to describe their peer grading system for MOOC-obsessed MOLB readers of all stripes. I have promised that there will be no rebuttals from me here. I simply asked Ben to come back later and tell us all how their system worked.

***

It is immensely difficult to create a massive open online course in the humanities and social sciences that approximates a traditional brick-and-mortar offering. This should come as no surprise given MOOCs’ origin (or to anyone reading Jonathan’s site). All three major MOOC platforms (Udacity, Coursera, and EdX) have their roots in computer science departments. And even their connectivist predecessors were created by academics in computing, too.

MOOCs emerged from the sciences because the sciences are scalable. They’re scalable on campus and they are scalable beyond campus. Sure, labs and more nuanced evaluation based on shown work are lost in multiple-choice examinations and task-based programming assignments, but the core forms of STEM assessments translate well to MOOCs. The humanities and social sciences, however, resist machine grading. Writing—synthetic expressions of the complexity of histories, societies, cultures, and creative works—rests at the center of evaluation in almost every single discipline and interdiscipline in the humanities and social sciences.

MOOC platforms are not ignorant of the centrality of writing in the humanities and social sciences, but their approximation of the campus experience—essentially relying on peer grading—has left much to be desired. I’ve never liked peer grading in my classroom and I think it’s even worse when also decoupled from instructor evaluation, as is the necessity with course enrollments regularly in the tens of thousands and instruction teams regularly less than half a dozen.

When grading is left to students, the results are mixed. Indeed, mixed may be a perfect term for peer assessments, since, as Jonathan has shown over at Inside Higher Ed, the quality of student grading varies widely. Efforts have already started to calibrate peer grading—Coursera has been a leader here—but this process takes a great deal of labor on the instruction team’s part and requires multiple offerings of the course before instruction team grades and student grades begin to correlate with much significance.

When I came to Online Learning in Penn’s School of Arts and Sciences early last summer, the first project that landed on my desk was to guide production on a new MOOC entitled History of the Slave South to be taught by history professor, Stephanie McCurry. I was excited by the assignment since my background is in the historical study of race, but I knew early on, too, that it was going to be a difficult course to construct. How would we keep the discussions respectful and on point? How could we teach historical skills in a class in which we could not assign any closed-access secondary sources? How could we assign essays without instructor or TA grades?

From the beginning of my work on History of the Slave South, that last question gnawed at me. How to construct an effective, enlightening peer review became one of two questions—the other being the translation of lab sessions to an online environment—I began to ask everyone I met with an interest in online learning. The months ticked away on the calendar and the January launch date approached. It was not until the Educause Conference in October that I found a promising lead.

There, Thomas Evans, Evonne Halasek, and Jennifer Michaels, all of The Ohio State University, presented on their experience with peer review in their MOOC on writing. This team had put a great deal of effort into constructing a peer review process that would benefit student learning, not simply replace instructor grading. And though they did not find a magical secret to perfecting peer assessment, they did find some promising directions to take the process in. The one that caught my attention was the success they found in description. Their peer review had students complete three tasks: description, assessment, and suggestion. In this describe-assess-suggest structure, they found high praise from students for the utility of descriptions their peers offered. It acted as a sort of mirror. It allowed students to see if they accomplished what they set out to accomplish. This focus on description immediately reminded me of a tactic I had used in my own on campus courses—the three questions of The Ignorant Schoolmaster.

Written by French philosopher Jacques Ranciere in 1987 (and first translated to English in 1991), The Ignorant Schoolmaster: Five Lessons in Intellectual Emancipation builds on Joseph Jacotot’s early-nineteenth-century panecastic pedagogy to argue for the “equality of intelligences.” Jacotot—the protagonist-muse of Ranciere’s text—was subdirector and professor at the École Polytechnique in Dijon and it was there where he crafted a pedagogy of “universal teaching,” which posited that each student is capable of teaching oneself as well as all others. We must free ourselves from the “masters” of traditional classrooms argues Jacotot and echoes Ranciere. Only then can we can achieve “intellectual emancipation.”

All a bit heady, no? And certainly a critique of expertise that does not sit well with historians like myself who reside in an institutional economy based so deeply on demonstrations of expertise.

For History of the Slave South, we did not leave behind the “master.” I cannot imagine a course like this without foundational lecture videos, highly curated texts, and refined discussion questions. Leaving students with no prerequisite knowledge of slavery’s history to teach each other is a scary proposition (though, a MOOC community might do better than “masters” in Texas). However, there is a place for Jacotot’s method in humanities and social sciences MOOCs—in peer review.

To teach his “students,” Jacotot asked them only three questions:

o What do you see here?
o What do you think about it?
o What do you make of it?

These questions, like the first stage of the Ohio State MOOC’s peer review, promote learning through description. This is an ideal method for a peer review in which the level of historical context about the subject at hand varies greatly from student to student. Our first assignment, for instance, tasks students with building tables from the Slave Voyages Database and interpreting them. We can’t assume whether or not a student knows if a peer’s interpretation is “correct.” Indeed, much of the point of the assignment is to expose the limits of interpretation in numerical representations of human lives. In essence, there is no correct answer. So rather than make our MOOC students into an army of pseudo-experts or TA proxies, we’ve tried to make their feedback more useful than judgmental. We’ve stopped calling it peer review or peer grading and, instead, termed it “peer reflection.” We want these reflections to act as a mirror to their peers. We want our writers to view their peers’ reflections as a way to see if they accomplished what they set out to accomplish. We significantly tempered the grading of our writing assignments so that it is now simply an “unsatisfactory,” “satisfactory,” “accomplished” scale. It is our hope—and here I must thank Stephanie and her teaching assistant, Roberto Saba, for fully committing to what I’m sure was a strange-sounding pitch with a radical nineteenth-century teaching method at its core—that a peer reflection system based in description will create a more useful experience for our learners.

After months of fretting and weeks of refining, Stephanie, Roberto, and I crafted the following instructions:

When assessing your peers’ work, please follow the instructions below:
1. Identify and describe your peer’s argument; what does it communicate to you?
2. Identify and describe your peer’s use of historical evidence; how does the evidence support the argument?
3. What makes your peer’s analysis persuasive? How could it be stronger?
To receive credit for your peer reflection, you must answer all three questions. Your reflection is required to be at least 150 words long and should not exceed 300.
In addition to this narrative feedback, please rate your peer’s essay as “unsatisfactory,” “satisfactory,” or “accomplished.”

We are under no delusions that our peer reflection will ever be as rich as most instructor feedback. But to claim humanities and social science MOOCs as anything even nearing an approximation of our campus offerings, we need to experiment with creative solutions in spaces in which massive, open courses have the greatest limitations. If we cannot find solutions to the conundrums that plague humanities and social sciences MOOCs soon, then the humanities and social sciences will once again falter where the STEM disciplines excel.

PS You can follow Ben on Twitter at @WigginsBenjamin.

About these ads

Actions

Information

10 responses

17 02 2014
Historiann

That list took “months of fretting and weeks of refining?” You could just have volunteered your time to evaluate the essays for McCurry’s MOOC instead. As we know, when MOOC consumers are actually required to read and write something, participation drops off dramatically, so even if each of you had a few hundred essays to evaluate it surely wouldn’t take you “months” or even “weeks.”

But then, MOOCs are a solution to a problem that doesn’t exist in humanities teaching. We have more qualified Ph.D.-holding humanists than can find tenure-track or even full-time employment. The problem with teaching the humanities is that people want it taught without the humans. And in the case of MOOCs, you’ll get exactly what you pay for.

19 02 2014
Benjamin Wiggins

Historiann,
I suppose I should have been more careful in my phrasing here. It was an issue we grappled with for months and, obviously, we didn’t spend weeks worth of time refining the assignment but rather refined it over a period of weeks. In other words, we cared about this issue from the beginning because we wanted to assign tasks and reflections on those tasks that would offer students a chance to practice historical practice. And we took an iterative approach to the instructions because precision is key when dealing with such a large audience from such a wide range of regions and linguistic backgrounds. I think it was a worthwhile investment of time and, if it seems to improve our students’ experiences with peer assessment, then perhaps it will work in other courses we produce.

Yes, completion rates–on average–drop when we assign more essays, exams, etc., but the workload required to gain a Statement of Accomplishment is not, to my knowledge, correlated with an increased participation drop off. Students are not blocked from participating in discussions, accessing the readings, watching the lectures, taking the quizzes, or even completing other assignments if they do not participate in peer reflections. We’d love to have all students do all the assignments, read all the readings, and view all the videos. However, there are many ways to engage with the course and we encourage students to participate even if they have no intention of “completing” the course.

You make an interesting point about labor in MOOC production, but I think your estimates are unrealistic. We will likely have between 2,500 and 5,000 submitted assignments by the end of the course. To grade, offer narrative feedback, and return those essays to students in the course of the ten weeks would be nearly impossible for our small team. Not to mention, I’m no expert in the U.S. South, and Stephanie, Roberto, and I each have many other commitments in our careers at Penn.

Thanks for your comments.

17 02 2014
Anne Corner

Very true that there are more humanities majors than jobs but people taking humanities courses still probably expect to learn something. I like the approach here as it gets away from “judging” others and puts the onus on what the peer thinks and knows. Still, it does just move the problem along. Now we need a second peer reflection to reflect on the reflection…..

18 02 2014
Doug didier

Ee
So yes I’m watching the lectures .

Get this off my chest..

Could be a lot easier. David blights civil war lectures at Yale website the transcript is a readable document . So I take the document .. Reformat in more or less outline format. Read it for content. Then watch the lecture. Try to pick up inflections in the words vice the words themselves.

Here , download text. Which is in segments.. The titles of the segments are links. The title of the lecture is in the syllabus. The files have different file types. I send some segments to a friend for transcription as can’t read the files Take snaps of the lecture with ipad.. And try to make document.

Then the forums.. Oh the forums. Each answer to the weekly question starts a new forum. How can I make any sense out of all of this? Some interesting stuff interspersed with random thoughts. If I take the time to work out an answer to the question what use is it to throw it in a trashcan?

So now I have a question or want to makes a comment on a particular segment. Can’t do it.. When macurry makes comment can comment to her comment either.

Do find the lectures pretty interesting. Would be interesting if american history were taught with a more slave centric thread. It seems to touch most things. Especially here in South Carolina . Been some pretty good books in the last ten years.

Professor is also an interesting person. Last spring Drove to williamsburg to hear her talk. Along with foner , glyph , ayers etc..

As for feedback or peer review. I would think your response here should be part of the course materials so students know what you are trying to accomplish. And I think that putting some papers up and doing an open source peer review would be more beneficial than this silo approach. Actually on a current MOOC .. The history and future of mostly higher education. Duke professor, forget her name, Davidson , she is pushing open source learning , she doesn’t know how to do it.. When I say do it I mean how to do it. Throw a bunch of technology in the hopper and see what comes out..

Ah that feels a lot better..

Nn

18 02 2014
j

I am really tired of humanities and other social scientist researchers thinking that STEM (which to my mind includes at least the natural science end of psychology and multi-field areas like cognitive science) is all about facts. Just like my mind has been opened by historians arguing that history is not about facts, science isn’t about facts either. Science is a method and a process. And assessing it via simplistic methods like multiple choice exams isn’t typical, or advisable. Do my students do as much writing as English and History majors? I’m quite sure not in their intro classes, but that is primarily because of size (my Intro Psych sections are well over 30, and that is at the school where I have multiple small sections instead of one big class of 120). Once they are into their research and methods sequence, there is a huge amount of writing, and also a huge amount of nuanced thinking that has to do with presenting and evaluating scientific evidence and arguments.

18 02 2014
Historiann

Great points, j. I think a lot of humanists rely on this lazy stereotype (and ignore the whole existence and point of LABS, duh).

19 02 2014
Benjamin Wiggins

Hi J,

I work a great deal with science faculty as my unit produces for-credit online and open, non-credit online courses in our School of Arts *and* Sciences. I’m aware of the limitations of MOOC assessments in the STEM disciplines, too, but I wanted to use this space to discuss the humanities and social sciences in relation to massive, online teaching. My juxtaposition is, of course, blunt. I’m happy to discuss the challenges facing STEM in massive, open courses offline (bwiggins[at]sas[dot]upenn[dot]edu) or here. I just think, in general terms, STEM courses are easier to transition to MOOCs relative to the humanities and social sciences ones.

Also, I see multiple choice questions as a complex form of assessment. Just because this form of assessment is machine gradable, doesn’t mean it has to only assess facts. Our medical school works quite hard at refining layered, complex multiple-choice assessments. The design of their multiple choice questions teach students how to synthesize a range of knowledges presented within segments and across lectures and other activities.

Thanks for your comments.

I really do appreciate your concern for a nuanced understanding of the STEM disciplines.

1 07 2014
Peter Onuf on MOOCs |

[…] The tension between lecturing vs. interaction is a major problem that MOOCs have so far failed to address in any meaningful way. Onuf admits that he stayed away from the discussion boards because “it’s embarrassing—it’s about me.” I can imagine that anyone would feel that way, but for a student in a face-to-face course, interaction with the professor is crucial. How much more so for a MOOC, where you are one of hundreds, thousands, or tens of thousands? If a student truly wanted to receive the full benefit of the course, s/he would need to have access to the professor. It’s one thing to lecture, even lecture well–it’s quite another to lead discussions, answer questions, moderate debates, and hold office hours. MOOCs seem inadequate for all of those necessities of a quality education. (For opposing views on peer evaluation, the backbone of MOOC “interaction,” read Jonathan Rees* and Ben Wiggins). […]

25 09 2014
Skills, Not “Stuff”: A Case for a New Kind of MOOC | Believe It or Not

[…] here – not all MOOCs have such poor feedback systems (check out an example of a better system here, which I’ll explain shortly) and not all learning occurs in MOOCs – it is possible that […]

30 09 2014
Is This True? | Allison Goldsberry

[…] Wiggins said: […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

Join 2,292 other followers

%d bloggers like this: