It’s ironic that of all the articles I’ve read recently that have tempted me to blog when I should be doing something else, it’s this one that’s gotten me to hit the “Add New Post” button. You see it’s about grading written essays, and that’s exactly what I should be doing right now. Actually, to be more specific, it’s about computers grading essays…badly:
The Babel generator, which [Les] Perelman [former director of undergraduate writing at the Massachusetts Institute of Technology] built with a team of students from MIT and Harvard University, can generate essays from scratch using as many as three keywords.
For this essay, Mr. Perelman has entered only one keyword: “privacy.” With the click of a button, the program produced a string of bloated sentences that, though grammatically correct and structurally sound, have no coherent meaning. Not to humans, anyway. But Mr. Perelman is not trying to impress humans. He is trying to fool machines.
Of course, the program succeeded beautifully:
Now, here in the office, Mr. Perelman copies the nonsensical text of the “privateness” essay and opens MY Access!, an online writing-instruction product that uses the same essay-scoring technology that the Graduate Management Admission Test employs as a second reader. He pastes the nonsense essay into the answer field and clicks “submit.”
Immediately the score appears on the screen: 5.4 points out of 6, with “advanced” ratings for “focus and meaning” and “language use and style.”
The point of this exercise is, of course, that an essay is more than just a string of grammatical rules. It represents an underlying idea, or, to be more exact, the students ability to express an underlying idea.
Who on earth then would turn automated graders (and nothing else) loose on their humanities students? Superprofessors, of course:
Daniel A. Bonevac, a philosophy professor at the University of Texas at Austin, is one of them. Last fall he taught “Ideas of the Twentieth Century” as both a MOOC and a traditional course at Austin. He assigned three essays.
He calibrated the edX software by scoring a random sample of 100 essays submitted by students in the MOOC version of the course—enough, in theory, to teach the machines to mimic his grading style.
The professor then unleashed the machines on the essays written by the students in the traditional section of the course. He also graded the same essays himself, and had his teaching assistants do the same. After the semester ended, he compared the scores for each essay.
The machines did pretty well. In general, the scores they gave lined up with those given by Mr. Bonevac’s human teaching assistants.
Big, fat, hairy deal. Wake me when the computer can write comments at the bottom of the essay. No student has ever learned anything from their letter grade. They learn from the comments. Indeed, as anybody who actually does this for a living knows it takes much, much more time to write comments on essays for precisely this reason than it does to read them. Even “A” students learn from the reasons the instructor put at the bottom of their essay as to why their essay got an “A.” That’s how they know what to do again next time. Anyone with any grade lower than that needs the instructor’s comments to know what to do better.
So what kind of philosophy class assigns essays that won’t be graded by human beings or will only be graded by unqualified peers? The same kind of philosophy class that doesn’t assign any required reading – which means an xMOOC. The name of my post describing the syllabus in that same philosophy MOOC is “What exactly does that certificate represent?” Without required reading or real grading of the required essays, it’s pretty clear the answer to that question is not much at all.
I might expect some confusion between writing and what it represents from students wondering whether MOOCs are right for them, but people who teach writing for a living, no matter what their discipline (but especially philosophy!), really ought to know better.