“[W]ith this bird everything is settled.”

The remains of the last known Passenger Pigeon at the Cincinnati Zoo.

“[W]ith a real nightingale we can never tell what is going to be sung, but with this bird [a mechanical nightingale] everything is settled. It can be opened and explained, so that people may understand how the waltzes are formed, and why one note follows upon another.”

- from Hans Christian Andersen, “The Nightingale,” 1844.*

I. The Game of Writing.

Last Thursday, the NYT ran an article about recent innovations in mechanical essay grading. You’ve probably read it by now, but you know the gist even if you haven’t. Geeks everywhere want to spare faculty the burden of grading student essays so that we can concentrate on other things. [Technology isn't just benign, it's good for everybody!] I always thought that grading essays was the thing that we humanities professors were supposed to concentrate upon, but then again what do I know?

As this blog has become all-MOOCs, all the time, here is the part of the article that I found most interesting:

Two start-ups, Coursera and Udacity, recently founded by Stanford faculty members to create “massive open online courses,” or MOOCs, are also committed to automated assessment systems because of the value of instant feedback.

“It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.

There is so much packed into those two extraordinary paragraphs that I barely know where to start.  When MOOC providers champion “the value of instant feedback,” my first question is “value to whom?” I do a lot of grading of written essays in the course my job, and I can tell you that the reason this process often takes so long is because I couldn’t possibly give instant feedback to students even if I wanted.

Good essay questions are about ideas. The essays students write should be about ideas too.  That means I have to sit and think about the ideas that students write in order to grade those questions. Instant feedback is therefore only a good thing if you think that writing assignments are something to get past rather than an opportunity for learning or, God forbid, reflection.

And then there’s that Koller quotation, one of a long series of quotes by Coursera’s founders that have continually left my jaw scraping the floor.  Suppose I ask my students to explain the historical impact of the New Deal.  What exactly is the “right” answer?  I always tell my students that I don’t grade on the basis of what their argument is, I grade on the basis of how well they defend it.  How is any artificial intelligence going to evaluate the inevitable issues of morality that good historical questions invoke from students?  It won’t, of course, and that should be a problem.

Perhaps more importantly, when students keep revising and resubmitting, who exactly are they trying to please? Programmers?  What do they know about good writing?  What values do they bring to the table?  Objectivity is not neutrality, as Thomas Haskell once explained.  As I write these words, this comment is at the top of the “Reader Picks” section of the comments under that NYT article:

Last year when my daughter was in 7th grade, her teacher started using computer essay grading. She would write her essay at home, using the computer, and would get a score. My daughter loves to write but got frustrated because the computer insited on correcting the grammatical errors of portions of the essay in which she used poetic language. In order to get a higher score, she begrudgingly changed her essay.

In short, computer grading destroys precisely the kind of creative thinking that writing is supposed to encourage.

Oh yeah, machines also aren’t very good at determining the accuracy of facts, which might be a problem in…you know…history courses.

II. Feedback Schmeedback

Reading that Koller quote also made me wonder exactly what kind of feedback students get when their essays are machine-graded.  After all, when I force students to play the game of writing , I make them write drafts.  On those drafts, I leave lots of comments. Those comments, in turn, serve as a guide to help students do better on their final papers.

So what kind of comments do students get back on machine-graded essays? Are they just blundering around in the dark?  That sounds a lot more frustrating than fun.  That NYT article suggests that the machines “provide general feedback, like telling a student whether an answer was on topic or not,” but what does that mean exactly?

In order to answer these questions, I did what any good 21st Century cyber-citizen does, I asked Twitter.  Follow that link through a long series of tweets and there are some excellent responses (to go with the inevitable less-than-140-character wisecracks).  Nevertheless, I still felt the need to dig deeper into this issue.

From what I can tell online, it appears that the big debate in the world of machine-grading is whether the scores that machines spit out match the same scores awarded by human graders.  Nowhere could I find anything about the machines giving comments, let alone comments that might actually prove useful.  It’s all about numbers, as if the quality of any piece of writing could ever be reduced to a single digit and a couple of categorizations.

Almost none of these computer science geniuses seem to understand that humanities disciplines are humanities disciplines because the answers to the kinds of questions we ask don’t have easy answers.  This is from Slate, published last year, discussing the problem of applying this technology to my actual field of expertise:

Compare and contrast the themes and argument found in the Declaration of Independence to those of other U.S. documents of historical and literary significance, such as the Olive Branch Petition.

Brown University computer scientist Eugene Charniak, an expert in artificial intelligence, says it could take another century for computer software to accurately score an essay written in response to a prompt like this one, because it is so difficult for computers to assess whether a piece of writing demonstrates real knowledge across a subject as broad as American history.

This may explain why Coursera offers peer-grading for one set of its courses, and is so enthusiastic about machine-grading essays for some others.  Indeed, doing this work I realized that the machine-grading problem is just about the exact equivalent of the peer grading problem.  They use these strategies because of the economics involved, not because they’re the best things to do for students.  That’s what makes quotes like this (from the same NYT article) so incredibly infuriating:

“One of our focuses is to help kids learn how to think critically,” said Victor Vuchic, a program officer at the Hewlett Foundation. “It’s probably impossible to do that with multiple-choice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time.”

Notice the slight-of-hand involved there?  Computer graders are much better than multiple-choice tests, not human graders.  Maybe they are, but who says those are our only two options?  As Mark B. Brown has argued, the fact that we’re even having this debate is an acknowledgement of permanent austerity.  In order to prevent professors and students alike from getting up in arms about this entire discussion, the MOOC enthusiasts and computer science geniuses that enable them have to redefine what education means.

III.  ”[W]ith this bird everything is settled.”

My goal as a teacher is to get students to decide for themselves what they think about history.  Do the proponents of mechanized grading even care about such things?  The kind of feedback that students get on machine-graded essays (or on peer-graded essays for that matter) suggests no.

As Mark Cheathem has strongly suggested elsewhere, machine-graded essays and scare tactics go together like wine and cheese.  ”You must automate everything!,” the profiteering vultures tell us, “Otherwise, the country will fall behind!”  [Isn't it really interesting that this strategy transcends national boundaries?  You'd think that the international professoriate could all just slow down together and keep ourselves employed, but I don't have my hopes up.]  If you think this argument is effective on seasoned professors who should really understand the concept of source bias better, imagine how effective it would be on undergraduates.

I can just hear the pitch now:  Don’t learn anything about critical thinking.  Critical thinking can actually impede your job prospects.  It’ll be just like The Organization Man all over again, only this time they’ll have studies to back them up:

[D]uring the great IT boom, the returns to cognitive skill rose.  Since then, the process has gone into reverse: demand for cognitive tasks is falling. Perhaps this is because installing robots consumes more resources than maintaining them, or perhaps it’s simply that the robots are doing an increasing number of those cognitive tasks.  But whatever the reason, we no longer want or need so many skilled workers doing non-routine tasks with a big analytical component.  The workers who can’t get those jobs are taking less skilled ones.  The lowest-skilled workers are dropping out entirely, many of them probably ending up on disability.

There are 115,000 janitors with college degrees in the United States.  Therefore, anybody who gets one must be a sucker.  Of course, not having a college degree will pretty much doom your chances of getting one of the remaining jobs that require critical thinking (and its corresponding pay level), but who wants to stand in the way of a newly emerging cliché?

What we do know is that cheapening education this way will assuredly put a lot of humanities professors (especially already-underemployed adjuncts without the protection of tenure) onto the unemployment line.  I say if we fall for these scare tactics and accept the values that mechanized grading represents, then we deserve to be there. Instead, we need to make the case that the skills we teach are important irrespective of how much  money students can earn by using them.  Kind of like listening to the song of a real nightingale.

Certainly, mechanical nightingales have yet to replace real nightingales out in the world. After all, they’re far too expensive. However, the values that the mechanical nightingale represents have done enormous damage to other bird species. Take the Passenger Pigeon, for example. Tens of thousands of those birds used to darken American skies:

Now they’re gone.  I, for one, feel like I’ve missed something, even if looking at a huge flock of birds has no commercial value.

In short, everything about the Passenger Pigeon is now settled – not in the same way that everything about the mechanical nightingale is settled, but settled nonetheless. Devalue critical thinking skills to the point that machines grading essays becomes acceptable and everything about education will be settled as well.  Our students will be settled like the mechanical nightingale is settled, singing the same song every time.  We humanities professors will be settled the same way that the Passenger Pigeon is settled, lucky if someone bothers to stuff us and display us anywhere since we’ll become forgotten relics of a bygone era.

But at least we won’t have to waste our time grading papers.

*  I am, of course, not nearly well-read enough to pick that reference out of thin air.  I got it from one of my favorite books of all time, Rebecca Solnit’s River of Shadows. Also, this post wouldn’t have been possible without the help of a slew of my tweeps, especially Cedar Riener, Mark Cheathem and Rohan Maitzen.

Everyone their own professor (w/ apologies to Carl Becker).

I. Everyone Their Own Sushi Chef.

Last summer, I tried sushi for the first time. I was in Korea at the Noryangjin Fish Market in Seoul and I went into a restaurant with a big banner in English that said, “Tourists welcome.” To that point in time, I had been one of those people who said smarmy things like, “I prefer my dead fish cooked, thank you.” However, having fallen under the spell of Andrew Zimmern ["If it looks good, eat it."], I thought it was time to give sushi a try.

Some of it was wonderful. I particularly remember the octopus sashimi because it was easily identifiable, and thanks to this scene from “No Reservations,” octopus had become the ultimate in creative Korean dining in my narrow American mind. Inevitably, some of the rest of the sushi I had there tasted awful to me. Unfortunately, the woman who served me there spoke very little English so I had no way to identify which kinds of sushi I liked, and which kinds I didn’t. I lacked guidance.

If I hadn’t had sushi, I never would have rented the film “Jiro Dreams of Sushi” on Netflix last week. That documentary about an 85-year-old sushi master, Jiro Ono, was simultaneously boring and riveting. [My wife lasted with it for only 45 minutes, but later she said to me, "I couldn't stop thinking about him [meaning Jiro] all day.”] Perhaps the most obvious takeaway from the film for a sushi novice like me is that preparing great sushi is a lot more complicated than it looks.

Towards the end of the movie, a food critic explains that eating at Jiro’s restaurant is a bit like listening to a symphony. The “performance” has movements and is scripted to the last detail. Patrons get one piece a time in a particular order to heighten taste sensations. The pieces that women receive are slightly smaller than those going to men so that their smaller mouths won’t slow down the production. The chefs are trained to see which hand each patron favors so that the pieces can be put on the side of the plate that each person favors too.

Why should anyone care about this? Sure, you can just eat sushi like I did, but don’t you want to make the most out of a new experience? If you understand sushi the way that Jiro does, you can learn more than you ever thought possible. The film (and this is what got my wife thinking so hard) even tells you something really special about the nature of work.

In order to see this subculture in all its glory, you have to have a guide. You have to get a sushi education.

II. Everyone Their Own Librarian.

When I was in graduate school, I used to play a game I called “Stump the Government Docs Librarian.” While I don’t think my dissertation was particularly good, it was well-researched in large part because I managed to find all sorts of extremely obscure reports that weren’t even in the US Government Serial Set thanks to the wonderful help I got at the State Historical Society of Wisconsin.

Obviously, Google Books and other such databases now make finding these kinds of obscure sources rather easy. One click, and all the greatest libraries of the world are at your fingertips. In the future, as these resources become even more powerful, they could actually put libraries and librarians out of business. As Nick Carr recognizes, this has created a certain amount of tension between the good folks putting together the Digital Public Library of America (DPLA) and local institutions:

The DPLA leadership is sensitive to this tension, sometimes to the point of defensiveness. In announcing his appointment, [DPLA Director Dan] Cohen wrote, “The DPLA will in no way replace the thousands of public libraries that are at the heart of so many communities across this country.” Yet the first sentence of the DPLA charter reads, “The Digital Public Library of America (DPLA) will make the cultural and scientific heritage of humanity available, free of charge, to all.” It’s hard to see how the DPLA will be able to fulfill such a broad mission without treading on the turf of local public libraries.

In this environment, librarians will have to provide a different kind of guidance. While we may no longer need them to help us find particular books, librarians can still help researchers figure out how to find needles in a series of gigantic haystacks. For example, what search term should you put in that database?

Besides interlibrary loan, I rely on local librarians to tell me what databases are available to me at our institution. I also need help learning how to improve my searches. As almost any history professor knows, students generally know almost nothing about how to search the web when a Google search of a single word or phrase does not yield usable results. That’s why I include lots of librarian time in every class I teach which requires a research paper. In fact, as the tech has gotten better, I’ve expanded that time rather than cut it back.

III. Everyone Their Own Historian.

In 1931, Carl Becker gave what may be the most famous presidential speech in the history of the American Historical Association. He called it “Everyman His Own Historian.” Sexist language aside, Becker’s speech was a poignant call for historians to recognize that academic history only has a purpose when it meets the needs of the public:

Berate him as we will for not reading our books, Mr. Everyman is stronger than we are, and sooner or later we must adapt our knowledge to his necessities. Otherwise he will leave us to our own devices, leave us it may be to cultivate a species of dry professional arrogance growing out of the thin soil of antiquarian research. Such research, valuable not in itself but for some ulterior purpose, will be of little import except in so far as it is transmuted into common knowledge. The history that lies inert in unread books does no work in the world. The history that does work in the world, the history that influences the course of history, is living history, that pattern of remembered events, whether true or false, that enlarges and enriches the collective specious present, the specious present of Mr. Everyman. It is for this reason that the history of history is a record of the “new history” that in every age rises to confound and supplant the old.

I tend to think of Becker’s speech most often during the periodic “Why can’t we all write like David McCulloch?” dust-ups that periodically echo through my profession. Nevertheless, Becker was no anarchist. He still envisioned a role for professional historians in a world where one did not have to have a Ph.D. in order to write good history:

[The history profession's] proper function is not to repeat the past but to make use of it, to correct and rationalize for common use Mr. Everyman’s mythological adaptation of what actually happened. We are surely under bond to be as honest and as intelligent as human frailty permits; but the secret of our success in the long run is in conforming to the temper of Mr. Everyman, which we seem to guide only because we are so sure, eventually, to follow it.

We historians, in other words, should provide guidance to people trying to come to grips with their own pasts. After all, nobody has the time to research everything they need to tell their own stories. Professional historians can provide the kind of analysis and perspective that amateur historians cannot or choose not to offer.

Without that perspective and analysis, not even David McCullough could write like David McCullough.

IV. Everyone Their Own Professor.

If nothing else, MOOCs [You just knew I'd get to them eventually, didn't you?] have brought the “Everyman His Own Historian” problem to every discipline in Academia. After all, why should I pay to go to college if I can listen to all the best professors in the world do their thing for free? In fact, if I run their lectures on 150% speed, I can learn everything I need to know in less time that it actually took for them to tell their stories in the first place! And I can do it at home in my pajamas! How can that not be progress?

Not so fast MOOC maniacs. Even the author of DIY U has noted that MOOCs aren’t an education by themselves. Take it away, Anya Kamenetz:

But I have something to say about MOOCs. Specifically about the quality of pedagogy in MOOCs as offered by platforms like Coursera and Udacity and edX. David Wiley, who has taught me a lot of stuff, said this at least five years ago, actually. MOOCs are content. Content is infrastructure. Infrastructure is just the first step.

MOOCs are content = a MOOC is not a course.

I suspect she and I would differ greatly on how much guidance a student needs after the MOOC starts, but isn’t it better to have more guidance rather than less? What too many people don’t understand is that the inevitable effect of MOOCs will be to take that guidance away entirely for most students.

This is what makes members of the MOOC Suicide Squad members of the MOOC Suicide Squad. When I read this post by Mark McDayter, I said to myself, “He’s solved the Cathy Davidson problem!,” namely how to deal with an educator whose goals you embrace, but whose methods will make those goals harder to achieve. You explain the political context in which those methods must inescapably operate:

Davidson, I am reasonably confident, does not support the gutting of Humanities departments and the replacement of teaching faculty with MOOCs. Indeed, she explicitly says as much. But her adoption of the language of the techno-enthusiasts is not nearly nuanced or critical enough to avoid giving aid and comfort to The Enemy.

Who is the Enemy? There are people who are enemies of higher education in general and people who are enemies of professors in particular. We will never win over enemies of higher education in general, who are often very conservative people who think that students can learn anything worth learning all by themselves with no access to professors at all. However, the enemies of professors in particular don’t always recognize that they are enemies of professors. Our job is to show them the light of reason.

If we professors can’t explain why the guidance we provide is an essential part of the college experience, we deserve the fate that inevitably awaits us.*

* That last link is to a Chronicle of Higher Ed article that’s subscription only as I write this, but you can still see my point here just by reading the headline.

Real college classes have writing assignments and required reading.

Do most MOOCs have required reading? I’ve been conversing with the proprietor of the blog Capitalist Imperialist Pig about that question in the comments here. They challenged me to look at all the excellent readings in two MOOCs, Gregory Nagy’s Ancient Greek Hero and Dan Ariely’s Coursera MOOC from Duke, so I did.*

Here’s the “Suggested Reading” statement for Ariely’s MOOC:

I will cover some of the material that is in my 3 books Predictably Irrational, Revised and Expanded Edition: The Hidden Forces That Shape Our Decisions (2008), The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home (2010), and The Honest Truth About Dishonesty: How We Lie to Everyone—Especially Ourselves (2012).

In other words, there’s no required reading, and one writing assignment which is (of course) peer-graded.

The Nagy course is better on reading. There’s a lot of free downloadable translations of Greek texts that you clearly need to read in order to pass the typical mulltiple choice lecture quizzes. But here’s the assessment and evaluation portion:

Students will be evaluated on assessment performance and participation. Assessments will be conducted each “hour” of the course. These will consist of quizzes on the reading (names, places, who is speaking to whom, etc.), as well as the application of principles and concepts central to the course.

Class participation, multiple choice quizzes and no writing at all.

Do you see a trend yet? Then consider this:

From #UTAustin #MOOC on 20th c. ideas: "While not formally required, prior to or during the course, you might wish to read the book" #USIH—
LD Burnett (@LDBurnett) March 30, 2013

Yes, that’s right. You can learn all about great 20th Century ideas at the University of Texas without having to read about any of them. And don’t forget about Harvard lite (law school edition)!

The trend should be clear now: MOOC providers don’t want to scare off potential students with too much work. Talk about teaching in a strait jacket! This is exactly why higher education should never be privatized in the first place. It degrades the quality of the product…a lot.

I’m sorry if this bursts anyone’s bubble, but watching videos on the Internet and maybe writing a few very short essays that the professor never sees isn’t college. Real college classes have writing assignments and required reading. Real college classes require access to the professor. To say MOOCs like these can somehow replace an actual college education is tantamount to fraud.

But that fraud isn’t just going to hurt students left with no access to college but for MOOCs. It’s going to hurt society in general too. This is from UW-Madison History Professor (Go Badgers!) Bill Cronon’s presidential address to the last American Historical Association convention (no subscription needed). [Warning: this story will make any real English or History teacher (and probably more than a few others) cry.]:

Still more poignant and worrisome was the young man who came up to me after a lecture I had just given at another university introducing the major themes of the very long book about Portage, Wisconsin, on which I have been working for longer than I care to admit. I sometimes describe that book as “Michener-length,” though that is a reference few students born in the past thirty years would recognize. So I usually add that I expect the final book to be at least five or six hundred pages long, covering as it does the history of this small Midwestern town from the glacier to now. The illustrated talk I give about Portage is intended to be a crowd-pleaser, with lots of engaging images and stories, and at the end of this particular lecture, a shy young man came up to say how much he had enjoyed it. I thanked him for his praise, but was then mystified when he added that he was very sorry he would never be able to read the book on which my talk was based. I sheepishly told him that although I was taking a long time to finish it, it would eventually be published, and he would certainly be able to read it then. He shook his head and said that was not what he meant. He reminded me that I had described the book as being more than five hundred pages long. Then, with a sad and embarrassed look on his face, he said he was simply incapable of reading such a book, that he had never in his life read anything so long. I was taken aback, but I am quite certain he was speaking in earnest, and that his regret was quite real.

Educating the masses without fixing the problem which that story represents isn’t really educating the masses, no matter what the MOOC maniacs tell you. At least we can blame the delusions of the venture capitalists on self interest. What’s the excuse of everyone in academia who should actually know better?

* These MOOC syllabus links require accounts from the MOOC providers in question. And yes, if you’re wondering, I did get an edX account in order to write this post. [The things I do for this blog!]

Update: A reliable source tells me that I screwed up in my quick read of Professor Ariely’s syllabus. Apparently, there is a substantial list of regular readings scattered around the Internet that I didn’t see. That would make his class more like Professor Nagy’s, a MOOC with a higher workload than most. My apologies to both Professor Ariely and CIP. In defense of my overall point though, I’ll note that the two Coursera world history courses that I’ve been directly involved with have no required reading at all.

Late Update: Since this post is getting a lot of late attention from people who are likely new visitors here, they might also want to read my follow up post on this subject, Some MOOCs are more inferior than others.

For their next trick, they’ll turn lead into gold.

“Will MOOCs Work for Writing?,” asks Chris Friend in Hybrid Pedagogy. The short answer is “No.” Two people who don’t know how to write cannot teach each other how to write. 200,000 people who do not know how to write cannot teach themselves how to write either. There is no such thing as a magic rubric. Teaching writing is labor intensive by definition.

Much to Friend’s credit he doesn’t exactly say “Yes, MOOCs will work for writing.” Instead, he suggests certain aspects of composition MOOCs that can make teaching writing easier anywhere. Unfortunately, the lessons he draws are exactly the kind of lessons that make MOOCs an inferior form of higher education. Due to my now well-known disgust for peer grading, I’ll concentrate on this one:

MOOCs make us rethink or reinforce our conceptions of assessment, pitting the allure and efficiency of mechanized essay grading using machine readers and database-driven plagiarism detection against the more intimate, yet slower, traditional human-scored high-stakes writing. The unsettling spectre of automated essay scoring, applied at large scales, calls us to refine our approaches to writing assessment. Noted composition-assessment scholar Ed White credits the Duke writing MOOC with including reflection, revision, peer review, and challenging reading as pillars of assessment practice, even though the instructor isn’t directly involved in the process. Given most instructors’ distaste for grading, refiguring its role in composition might be the best thing we take from MOOCs.

So just because something more awful than peer-grading exists, we should give up on the idea that every student deserves access to the professor? How cowardly. It’s obvious from the rest of the post that Friend cares deeply about effective education, but the “third way” on MOOCs that he proposes (somewhere between loathing and irrational enthusiasm) is a recipe for disaster.

While some of us debate the possibility of whether the impossible is actually possible, accepted standards are gradually changing so that every aspect of higher ed can be automated, whether or not that’s a good idea (and if you care about quality, it almost always is a bad one). God forbid if we actually start bringing peer grading (as opposed to just peer evaluation) into the physical classroom. Then it will be even easier for the powers that be to eliminate professors entirely as the difference between face-to-face instruction and a MOOC would then actually be minimal.

With all due respect to all the lovely people who are trying to use cutting edge technology to make composition (or other writing-based) MOOCs possible, I don’t need to conduct an experiment to know whether you can turn lead into gold, even if Bill Gates wants it to be so. And while the MOOC enthusiasts continue to fiddle, Rome burns on.

Bang your head (Peer grading edition).

While most people here and elsewhere seem to have appreciated my essay about the futility of peer grading from Inside Higher Ed last week, I have seen enough serious critiques that I want to defend myself here.  While I certainly understand why any superprofessor would want to teach the best MOOCs they can teach, I nonetheless offer what follows as a reality check.

Debbie Morrison, who is open-minded enough to read this blog even though it might well be the polar opposite of hers, rebuts my argument by citing research which suggests the circumstances under which peer grading can be effective:

1) When learners are at a similar skill level.

2) When assignments are low stakes [i.e. when a course is taken for professional development of personal interest...]

3) Where credit is not granted

4) When learners are mature, self-directed and motivated.

5) When learners have experience in learning and navigating within a networked setting [if the review is completed in an open and online setting.]

6) Learners have a developed set of communication skills.

The breakdown in peer grading occurs when the learning environment cannot provide the conditions as mentioned above.

Now that’s all well and good, but the sheer massiveness of a MOOC combined with Coursera’s obligation towards its investors to eventually turn a profit pretty much assures that every single one of those conditions will be violated at one point or another.  More importantly, the rich university administrations that produce course content as well as the poor university administrations that long to replace their faculty with videotaped superprofessors and poorly-paid teaching assistants have every incentive in the world to break every one of those conditions too.

The other critique I particularly appreciated appeared very late in the week at the bottom of the comments to my original article.  Its author signed in as “EnglishTeacher,” and went through the same peer grading process I did as a student in Jeremy Adelman’s course.  They write:

“No one has ever claimed that MOOCs do, can or should replace the full learning experience available to those fortunate enough to be able to be in an engaged on-campus classroom.  And no one has ever claimed that students can take the place of humanities professors.”

Actually, Daphne Koller of Coursera just claimed that students can do a BETTER job at grading essays than humanities professors about a week and a half ago.  With respect to taking over the rest of any particular professor’s job, if the superprofessor provides all the content and student peers do all the grading, what exactly is left?  Not bloody much.  Yes, we can go from desk to desk like my high school math teacher used to do while we worked through our algebra problems, but what kind of wage is that going to get us (particularly as most professors don’t have union representation like so many secondary school teachers do)?

I admire everyone who wants to experiment with new technology to make higher education better for their students.  I really do.  Unfortunately, while those people bang their heads against a wall as part of a futile quest to build a better mousetrap than the one we already have, the powers that be will still be doing their best to make us all technologically unemployed whether robots can do our jobs any better or not.

In the end, the value of peer grading comes down this:  Who can do a better job at grading students essays, peers or professors?  If the answer is “peers,” then why do professors exist at all?*  If the answer is “professors,” then why are so many people wasting their time trying to figure out a way to make the wrong answer right?  As David Golumbia has explained:

MOOCs are being deployed specifically as part of an economic argument whose consequences for liberal arts education are designed to be explosive: they are designed to make liberal arts education emerge as too expensive for us to afford.

Peer grading, like the MOOCs it facilitates, is designed to make the unacceptable acceptable.  It is a strategy created to fit the contours of permanent austerity rather than for the benefit of our students.  So while I agree that pigs look better with lipstick on them, that doesn’t mean the pig becomes any less porcine.

Should anyone choose to keep banging away at the peer grading problem anyway then be my guest.  Just remember that you have been warned, not just about your prospects for ultimate success, but also of the larger political context in which that banging must inevitably occur.

* By the way, I expect the resignation letters of all humanities professors who answer this way to be tendered as soon as they have time to pack up their offices.

A tool for a professor-centered edtech world.

This one’s for all the historians out there, and perhaps all my readers who still think I’m a Luddite. Because this where I want to tell you about something I’ve been working on and see if any of you out there want to help.

I’ve mentioned Milestone Documents here many times before. A few years ago, when I decided to teach my American survey class without a textbook I started using their service as a replacement. For a $20 semester long subscription, students get access to a wide variety of online primary sources, all well-introduced and edited down to a size that most college freshman will actually read. What I like most about teaching this way is that my assigned reading perfectly compliments what I actually teach. I just go down their list of documents and link to each one I want students read on my syllabus.

While I was delighted to ditch my textbook, a lot of other professors have been reluctant to do so. Therefore, Milestone Documents has begun the process of putting together secondary materials of their own, something they are calling a “textbook layer.” However, this one will be like no other textbook available. It’s going to be written in discrete pieces, like their documents collection. That way professors can assign the parts of the book they teach, and ignore the ones that they don’t. Of course, those pieces will be organized by themes. The professors who assign them will be able to pick not only the parts of the textbook that they teach, but the appropriate primary sources too. Best of all, access to the textbook layer and the original Milestone Documents will cost only $30 per student per semester, just $10 more than access to Milestone Documents by itself.

I’m the Editor-in-Chief of the US History II textbook subject area and need help on two fronts: Milestone Docs needs authors to write the individual textbook entries and review board editors to help make the whole thing coherent. Indeed, while I’ve been calling this a textbook, from an editorial standpoint this process will bear a greater resemblance to a big encyclopedia. Since I’m not the employer I can’t talk payment rates, but I can assure you that compensation for authors and editors will be competitive with other encyclopedia projects out there.

So if you specialize in post-1877 US History and are interested in joining us, e-mail a cv and short note to me at the address at right and cc it to Neil Schlager, CEO of Milestone Documents (Neil [at] milestonedocuments.com). With respect to the editorial review board team, we’re looking for people of various specialties who have substantial teaching experience at the college level, but grad students are more than welcome to join us as writers. If you specialize in other survey areas, including US History to 1877, World History or Western Civ, send your cv directly to Neil and he’ll put you in touch with the Editor in Chief for the appropriate area.

A dull, wonkish post about technology and historical research.

Today I’m starting the second (and last) week of my research trip in and around DC. Last week I was at the Library of Congress. After I leave this Panera, I start in at Archives II, assuming I can figure out the bus schedule.

Most of the papers that I’ve been looking at have turned out to be obscure published speeches and government reports. As I don’t have much time here in the great scheme of things, I’ve found myself checking Google Books to see if everything in front of me was already there. Some of it was. Some of it wasn’t.

I can’t tell you what an enormous change this has been for me. Normally, I’d be taking everything I’m sure I’d use eventually up to the copier and copying like crazy. Now, the LOC has put in the best book scanners that I’ve ever seen (both in the Manuscript Reading Room and in Adams) which you can use for free as long as you have a thumb drive to store your results. Free stuff from the government! Somebody tell Paul Ryan!

I scanned a few pamphlets, but for visual material it’s just priceless. Indeed, I’m also deep in the throes of picking pictures for this book and I’ve basically had to go by memory. Next time it’s going to be different. Of course, this is all to save the spines of the books, but I still feel like I hit the lottery.

I may be alone in this feeling because it seems that most people have gone entirely to cameras. I’ve seen some mounted on tripods, while some people it seems just bring in the same $90 Samsungs that they use on their vacations and snap away. I even saw one woman snapping pictures of her microfilm reader. [It may sound logical, but it sure looks funny.]

I was prepared to start using a camera and a tripod before this sabbatical started, but a few weeks back I changed my mind. Most of the sources I’m using using are pre-1923 and published. Indeed, I really can download most of them for free with a lot less hassle.

For the one-of-a-kind published and archival material (and you know, as more books are becoming available to everyone this is what will set great works of history apart in the future) I’ve got Zotero, which still beats the heck out of note cards. This is going to be my first all Zotero book, and I’m certain it’s going to take months off the writing process. After all, it’s having intellectual control of your research that’s most important when technology makes it easy to flood you with material.

Perhaps it’s a sign of my advanced age that I’ve chosen to avoid swimming in document snap shots. When I do research, I actually enjoy reading what I find in folders rather than just snapping them or even trying to copy it all. In fact, I think I do some of my best thinking that way. And in case you didn’t notice, I hate reading off a computer screen when it can be easily avoided.

Is it any wonder then why I don’t want my entire job to turn out that way?

“[Y]ou must cut down the mightiest tree in the forest with…a herring.”

Laura Gibbs deserves some kind of prize for public service. I’ve been tweeting her series of posts about peer grading in Coursera for a while now, but since Audrey Watters has written them up I figured I might as well consider them here too. What you need to know going in is that Laura teaches online for the University of Oklahoma so she’s clearly rooting for Coursera as she takes their course on Science Fiction and Fantasy. I think this makes her indictments of the process all the more damning.

For example, there’s this:

So, what kind of data is Coursera collecting about the efficacy of this process? None. What kind of feedback are people getting on their feedback? None. What kind of guidelines and tips did we get on offering feedback? (Almost) none. Given that this is a skill, and a skill that many people have not had to use in the past, I think we would need a LOT of tips and guidelines to help with that, along with feedback so that people who are just now developing this skill can estimate how well they are doing.

And this:

By far the biggest problem, though, is vague and/or inaccurate feedback… and that’s a much harder problem to solve. It’s much like the problem with the poor quality of the essays overall; yes, there are inappropriate essays (blank essays, essays only a few words long, plagiarized essays, even spam essays) that need to be flagged – but the larger problem is the bewildering number of essays that are of such poor quality that it gets very discouraging to spend time on them. Without some kind of additional instructional component to the class, I am just not convinced that this often unreliable and/or unhelpful anonymous peer feedback can really help people to improve their writing.

Remember, this is just about the peer feedback system. I haven’t even mentioned the plagiarism problem or the lack of writing instruction in general.

“Can peer feedback really work in a setting where there is so little community and where this is little sense of reciprocity?,” asks Audrey. Well, that depends upon how you define the term “work.”

If you watched that Daphne Koller TED video, you probably remember the joke about how she tried to convince those terrible humanities professors that multiple choice was a perfectly acceptable way to test for higher order critical thinking and they did’t buy it. Ha ha ha. Unable to do that, they went with Plan B: peer grading. The impression this story left on me was that Coursera was only interested in doing the absolute minimum in order to make their humanities classes acceptable. Certainly, everything Laura has written suggests that they didn’t exactly put much forethought into some pretty basic problems.

But I want to take this point one step further. I would argue that creating an effective peer review process for grading writing is impossible – like chopping down the mightiest tree in the forest with a herring. Since writing is a skill that you never really stop learning, peer grading is therefore almost always the blind leading the blind.

For example, I am about to go into deep seclusion to polish my book manuscript for the last time before it hits the copy editor. It needs polishing because I have a bad habit of using the passive voice the first time I write anything at all complicated. Usually I turn those sentences around when I catch them during proofing, but I don’t always catch them. If your peers don’t know what passive voice is, or (as seems very likely in a lot of these Coursera classes) your peers don’t even speak English as their first language, learning how to write well solely from them is going to be impossible.

Since I teach history, I am prone to think of learning history as an excellent end in itself. However, if you desire employment when college is over, learning how to write well is the best skill that academic history classes can offer you. No wonder employers don’t take job applicants with online college degrees seriously then.

It appears that Coursera is giving them little reason to think otherwise.

So I got a book contract…

…which explains why I want to write about publishing academic history from the author’s point of view. Normally, I save my history blogging for the nice folks at the Historical Society, but this is going to be too personal for there. However, I promise what follows will be more than just a victory lap as I think I’ve actually learned a few things during this very long project which I want to share.

Seven publishers rejected my revised dissertation. My evenlual publisher required me to buy ninety-five copies (at $34.95 a pop that’s quite a hit) in order for them to put the book out. This alone was enough for me to get tenure, so it was a good investment. Still, I was deeply ashamed for a long time even after I found out that this has become increasingly common practice in academic publishing.

I started the book I just got a contract for in 2000, while I was waiting for something to come together on the dissertation. [Yes, you read that date right, but to be fair I have published two other books in the interim, including one you can pre-order right now.] This newest one is a history of the American ice and refrigeration industries, tentatively titled Refrigeration Nation: How America Learned to Control the Cold.* The publisher will be the Johns Hopkins University Press. The Johns Hopkins University Press was one of the seven publishers that rejected my revised dissertation. Since the manuscript is done and has survived peer review, I suspect it will come out sometime early next year.

So what changed between then and now?:

1) Topics matter. My dissertation was on labor policy in the American steel industry. It’s an interesting subject, but it is hardly empty historiographic space. In contrast, the last monograph on refrigeration in America came out in 1953. Even then, that book was a business history. I spent years reading old trade journals so that I could learn the details of different refrigeration technologies. As a result, my book should be close to unique. For a technology this historically important, I still find that fact amazing.

2) Learning by doing. While there was merit in my dissertation (particularly the research) I wrote myself into a corner early on and couldn’t get out. What does that mean? I structured the entire book before I finished my research and couldn’t adapt it well when the peer reviews came in. To make matters worse, I kept trying to change that manuscript to meet the demands of the last review anyway. This time, I took lots of time to decide exactly how I wanted to organize the manuscript, and tried to be flexible. I think this is one of those lessons you just have to learn by doing it wrong the first time so that you never forget its importance going forward.

3) I took it on the road. Some people think that conferences are just glorified vacations. Sure, we all sneak out for a few hours at some point during the weekend sometimes to see something historical, but I’ve been giving parts of this manuscript as conference papers since around 2006 and I can’t tell you how much that’s helped. A Hagley conference in 2006 is where I figured out my argumett for the first time. It was at a food studies conference at UC-Davis in 2009 that I realized that I needed to include a global perspective in order to demonstrate the fact that America is particularly refrigeration crazy. That took about three extra years, but it’s a much better book as a result. Most importantly, it was at the Society for the History of Technology meeting three years ago that I got recruited by the JHU Press straight out of my session. All they knew was the subject of the book and the contents of the paper I had just given, but that proved to be enough (See change #1).

4) Arguments matter. The problem with my dissertation was that I wrote it story first, argument afterward. As a result, my argument was overly simplistic. When I started this project, I actually thought describing a dead industry (I started with just ice, but then went forward to the present) would be enough to justify publication. That might be true if the book were just for refrigerating engineers, but it’s not. An argument shouldn’t overwhelm a story, but it needs to be there nonetheless in order to make your expertise interesting to people who wouldn’t care otherwise. It’s sad that I didn’t figure that out while I was still in grad school, but I’m glad I at least figured it out now while I’m still young enough to have a few more books in me.

Speaking of more books, I get to use my sabbatical this semester to start a biography of Harvey Wiley, the guy who was the first head of what would eventually become the Food and Drug Administration. I want to use his life as a way to discuss the adulteration of foods of all kinds. This will include meat, bread, sugar, whiskey – even Coca-Cola.

Continuing my uphill climb, my plan this time is to get an agent and a contract with an advance from a trade press. If you happen to have an agent – or even better, if you happen to be an agent – you know how I write if you read this blog regularly. Any interested parties should use my contact info on the right.

PS Regular snarky technoskeptical blogging will resume with my next post.

* Yes, I know the “nation” title formulation is now cliché, but it really fits the argument. Also, it rhymes! How many other historians do you know have books with titles that rhyme?

“Has he lost his mind?”

So I signed up for a MOOC. Seriously. A History of the World since 1300, taught by Jeremy Adelman from a certain university located in my hometown of Princeton, New Jersey.

Why would I of all people do such a thing? Well, I’ve had something of a complex about my overspecialization in American history since my first teaching job at Whitman College. Unlike Wisconsin, which had Americanists coming out of its ears, Americanists were in the minority at Whitman so the old Europeanists teased me for having such a limited knowledge base. I’ve rectified that somewhat through independent reading, but I could definitely stand to learn more specific factual knowledge from outside my country of specialty.

Then I watched this TED talk by Coursera’s Daphne Koller and got a little excited. I had never seen so detailed an explanation of the mechanics of MOOCs, and it seems as if they’ve gone to great lengths to help students learn the kind of factual knowledge that I’m missing when it comes to world history.

Have I lost my mind? Nope. Am I pulling a Whittaker Chambers or a David Horowitz on the subject of MOOCs? Nope. As anyone who’s ever watched a TED video knows, there are parts of every such speech that make you want to take a hammer to your computer screen (and I’ll get to that one for me in this speech in just a second). However, as I’m on sabbatical for this coming this semester, learning world history seems like a good use for some of my extra time.* In fact, there’s a place on my annual performance review for extra education which I’ve never had occasion to mark before. I’m absolutely going to put this down.

So what’s the problem? Well, for starters the course has only one text and even that’s only recommended. Is there a history class anywhere in America (let alone Princeton) which has no required reading? Seriously, I have a question for all the education geniuses out there who want me to flip my classroom: When are students going to do the reading I assign them? After all, history is a literary art, not a trivia game.

Now here’s the part of that Daphne Koller video that came close to inspiring me to violence (my transcription):

“Well, of course, we cannot yet grade the range of work one needs for all courses. Specifically, what’s lacking is the kind of critical thinking work that is so essential in such disciplines as the humanities, social sciences, business and others. So we tried to convince, for example, some of our humanities faculty that multiple choice was not such a bad strategy. That didn’t go over really well.

[Audience chuckles]

So we had to come up with a different solution. And the solution we ended up using is peer grading. It turns out that previous studies show, like this one by Sadler and Good, that peer grading is a surprisingly effective strategy for providing reproducable grades. It was tried only in small classes, but there it showed, for example, that these student-assigned grades on the Y-axis are actually very well-coordinated with the teacher assigned grades on the X-axis. What’s even more surprising, self-grades, where students grade there own work critically – so long as you incentivize them properly so that they can’t give themselves a perfect score – are actually even better-correlated with the teacher grades. So this is an effective strategy that can be used for grading at scale and is also a useful learning strategy for the students because they actually learn from the experience.“

I’ve covered this precise subject before, but this sounds even worse to me now than it did then. When testing becomes the be all and end all of American education at all levels, we act like it’s OK to care only about the math and not about actual learning.

How are students ever going to learn anything about critical thinking in any subject without good, thoughtful comments? The students are incentivized to get done with their peer grading as soon as possible because it’s not their grade. When I grade, my salary incentivizes me to actually explain to my students how to do better next time. As further incentive, when my comments actually help, it makes grading their papers easier in the future. That kind of attention will never scale up. Period.

I only worry if anyone will care. I guess I don’t care for purposes of what I want out of this class, but presumably I know something about critical thinking already.**

* No lazy professor jokes, please. As anyone who’s ever been on sabbatical knows, it’s not a work-free period. It’s a period when you do different kinds of work. I’ve been telling people that I’ll be a professional writer until January. I have a new research project to work on, but of course I’m going to write about taking this course too.

** If I can’t ace this course I’m going to be so ashamed.