“[A]nd the number of the counting shall be three.”

While I was making my way home from Atlanta on Sunday, a whole bunch of my virtual and actual friends were still at the Organization of American Historians annual meeting discussing whether blogging is scholarship. While I’m sorely tempted to weigh in on this question myself, I think I’d rather follow Mike O’Malley’s example and consider exactly what scholarship is. Or to put it a slightly different way, what and who is scholarship for? Or maybe just why scholarship?

What’s sent me down this path before I even saw O’Malley’s post is this rather amazing article from Smithsonian (which I found via Rebecca Schuman, who’s probably still laughing her ass off about this days after she first read it):

“There are a lot of scientific papers out there. One estimate puts the count at 1.8 million articles published each year, in about 28,000 journals. Who actually reads those papers? According to one 2007 study, not many people: half of academic papers are read only by their authors and journal editors, the study’s authors write.

But not all academics accept that they have an audience of three. There’s a heated dispute around academic readership and citation—enough that there have been studies about reading studies going back for more than two decades.

In the 2007 study, the authors introduce their topic by noting that “as many as 50% of papers are never read by anyone other than their authors, referees and journal editors.” They also claim that 90 percent of papers published are never cited.”

Of course, the flies in the ointment of this discussion are tenure and promotion standards. Early-career scholars with blogs want blogging to be scholarship because that will make tenure easier to attain. I know that sounds bad, but really what’s the use of running the normal academic peer review gauntlet if it’s likely that only three people will read the result?

Coincidentally, this discussion and this article happened at the same time that I have to worry about precisely this sort of thing once again. Yes, I’m a tenured full professor, but as anybody among the somewhat more than three people who read this blog regularly know our administration here at CSU-Pueblo is trying very hard to move the vast majority of professors at this institution from a 3-3 (or 9 credit) to a 4-4 (or 12 credit) teaching load. While I was once optimistic that there would be enough exceptions to that standard that most active scholars on campus would be able to avoid it and continue their research apace, I am not anymore.

Here’s why: A few weeks ago, our Provost published his new research standards at the back of a grant application form for a single semester of release time. To my knowledge, he did not consult our faculty senate or any faculty members whatsoever before doing so. Here is a selection from that document (no link because it was e-mail only, e-mail attachment only to be exact):

“At CSU-Pueblo, faculty are expected to teach 12 credit hours per semester (and engage in research/scholarly/creative activity, and perform service). I emphasize that regular scholarly activity is expected of faculty who teach a 12 cr hr teaching load per semester. Awarding equivalency time to conduct research/scholarly/creative activity, above and beyond the usual expectations that we have of faculty, requires careful justification – even moreso at a public institution, in an environment with significantly constrained resources.”

Here’s what it says about release time for scholarly activity in our faculty handbook:

“After consultation with the faculty and Chair of a department, the Dean shall recommend to the Provost all requests for release from teaching. Faculty members released from teaching assignments shall devote a minimum of three (3) clock hours per week for each semester hour of released time to tasks associated with such release….Release from teaching to engage in sponsored research, University supported scholarly or creative activity, University service or other approved activities may be authorized by the Provost dependent upon the availability of funds and program needs.”

In other words, we’re going from an environment in which the vast majority of faculty members received that one course release to an environment in which we all have to prove that we’re not ripping off the taxpayers of Colorado and we still might not get that course release anyway. Furthermore, there’s been no hint that the standards on our annual performance reviews will be amended at all to reflect this rather significant change in policy.

While I’m fortunate enough to have no need to submit this blog as proof of scholarship, other faculty members on campus might not be quite as productive as I’ve been lately. Here’s the gauntlet that we all have to run to get one of 20 or so release time “fellowships” to pay for our adjunct replacements (as described in that policy statement I referenced above):

“The Provost will not approve equivalency time for research/scholarly/creative activity for Fall 2014-Spring 2015 if there is not a demonstrable peer-reviewed work product within the previous 2 or 3 years, depending upon the amount of equivalency time requested.”

It so happens that I approve of the peer review process. In most cases it has significantly improved the work that I’ve published, but as anybody with actual experience in peer review knows this slows things down to an unimaginable degree. For example, I wrote on article to mark the centennial of the Ludlow Massacre for Labor during my sabbatical a year and a half ago in order to make the anniversary itself, which is this very week. It’s accepted, but won’t be published until the fall, months after the anniversary is over.

Will more than three people read that article? Labor is a very good journal so I think so. However, even before I read that Smithsonian article I had become increasingly convinced that most academic journals are utterly useless. The value of blogging (or God forbid practicing actual journalism) is that you’re almost instantly guaranteed a much wider audience than publication in even the most respected academic journals will ever give you. Shouldn’t the point of scholarship be to influence the way the world works? If so, how can anybody justify a narrow fixation on peer review if almost nobody reads the results?

What troubles me most, however, is my administration’s demand for a “demonstrable peer-reviewed work product” within a two to three year window. My last book took me (on and off) thirteen years. Nevertheless, I still want to write more books. Not only that, I want to write more books that people will actually read. I’m currently close to being under contract to write two more comparatively quick refrigeration related books using my surplus research. Both will be peer-reviewed (or at least extensively peer-edited). After that, however, my Harvey Wiley biography is going to take a huge amount of time for me to finish because his papers are all back East and that extra class I’ll be teaching starting this fall isn’t going to speed that process up any.

As you might imagine, this whole situation makes me incredibly sad. If the only solution to this problem is to write short, crappy, purely academic work that reads like the instructions for the Holy Hand Grenade of Antioch and only three people ever read it, I don’t know if I want to play this game anymore.

Advertisement

MOOOOOOOOOCS!!!: #AHA2014 edition.

So we did our panel yesterday. The nice people at the History News Network (aka David Austin Walsh) did indeed film it, but they say the tape won’t be online until next week. In the meantime, I thought I’d just dump my (probably not all that accurate) notes from the presentations online while I’m breakfasting at Starbucks on my way to the library again, with just a couple of introductory remarks of my own for each speaker:

1) Philip Zelikow

For those of you who may have seen any of his MOOC, Philip really is that poised. He says he did multiple takes while filming those lectures, but I’m telling you he couldn’t have needed all that many. I was so grateful to him for coming because (for reasons I’ll explain below) without him most of the session would have been like the sound of one hand clapping:

MOOCs different from regular online courses.
14 weeks (the length of both his and Jeremy’s course) is highly unusual.
UVA does MOOCs for outreach.
“These [meaning MOOCs] are not cheap” if you want to do them well.
Huge #s are meaningless. How many people actually try out the course?
His course had 94 separate video segments. If the narrative arc worked some were as long as 30 mins.
90,000 people signed up.
15,000 people gave the course the old college try.
10,000 stuck with it.
5,000 of those were online auditors (weren’t taking the tests).
2,500 more were downloading and watching the segments offline.
“Most gratifying teaching experience of my entire life.”
Someone even sent him flowers [Unsolicited, of course].
This was an overload. As a dean, he isn’t even supposed to be teaching at all.
Advantages of MOOCs:
1) Allows for more elaborate integration of media.
2) Students can freeze on maps.
3) Students can learn at different speeds.
4) His students get the follow-up explanation from him, not the TA.
“In a way, I’m doing the TA sections.”
TAs are doing history labs, something new that wouldn’t happen otherwise.
Students self-report that they work 50% harder in his flipped course.
Is the pure online material useful? Yes, students in and out of UVA said it was highly enriching.
Flipping class is highly satisfying for everyone involved.

2) Me

Well, of course I couldn’t take notes on my own paper and (as I explained earlier this week) I can’t post it yet. For now, I’ll just say this: labor historian at heart that I am, I made the “MOOCs as scientific management argument,” comparing the “unbundling” of the historian’s job in the classroom to having Frederick Taylor stand behind you with a stopwatch. Break up any job, and you can pay the people performing the component parts less – often much less. The folks who were livetweeting me seemed to think I was being combative, but I really do understand that nobody interested in MOOCs welcomes the virtual equivalent of academic Taylorization. My fear is that if we really do leave everything to the market, we’ll get this outcome nonetheless.

3) Ann Little

It really is a privilege to be able to hang with Historiann in the non-virtual world. For those of you who have never met her, she does indeed talk like she writes in the sense that she is both incredibly astute and hilarious at the same time. Funny story: I was going to avoid using the word “superprofessor” with two of them in the room because I thought it was too inflammatory, but Ann just dove right in. Therefore, I lapsed repeatedly by the end of the session too:

What MOOCs can’t do well:

1) MOOCs obscure the real work of teaching.
2) MOOCs make it difficult to teach controversial content.
3) Compares the upcoming MOOC and in-class version of Stephanie McCurry’s slavery class at U Penn. [The MOOC version is much easier.]

Professors face attacks on their politics first to soften us up for later budget attacks.
Quotes my Provost on us only working three days a week.
Will superprofessors be too afraid to cover race?
Students need to hear other people talking about controversial material.
Given a say in the matter, would students only pick Whiggish studies of progress?
Would polarization occur in a gender or sexuality class?
Would the discussion boards in a class like that look like the angriest corners of the Internet in general?
How will McCurry monitor the shocking nature of the material on the discussion board?
McCurry at Penn: Six books, primary sources and discussion section.
Her MOOC appears to have nothing but online primary sources.
Real education requires skin in the game from both sides, both the teacher and the student.

4) Jeremy Adelman:

Jeremy’s plan was to take Amtrak down from Princeton to DC in the morning, but there was that big snowstorm in the Northeast the night before. I was getting optimistic travel updates from him by e-mail the night before, but they kept getting more pessimistic as the reality of Amtrak under stress began to sink in. I was kidding him about making a dramatic entrance from the back of the room, and it turns out that’s exactly what he did – straight from Union Station, about 25 minutes before the end of the session. I don’t think he even took off his coat before he started talking:

“My attitude is experimental”
Starts with the fact that he doesn’t want to ban students taking notes on laptops.
Didn’t want to have an adversarial relationship with his students.
[This explains his interest in the flipped classroom.]
Interested in the promise of global learning.
His hope was the world could talk to itself.
Papers were the part of the MOOC that worked the least well.
Forums worked in the sense that there was global dialogue.
The problem was that there was less dialogue between Princeton students and the world.
Second version of his MOOC had Princeton students blogging for the global audience.
Princeton students didn’t want to go out and engage the planet.
From flipped classroom at Princeton, students got a lot out of lectures for the first time in 25 years.
Most MOOCs give certificates. We gave nothing. Going to change that.
Lifelong learners take relatively passive attitude toward learning.
Going through college without learning how to write is a problem.
Jeremy also announced (quite offhandedly) that he’s leaving Coursera.

That’s it for me for now. Hopefully, I’ll get some time to write something more reflective about this next week, but my semester starts a week from Monday and I still have syllabi to write (during those three days that I’m actually working).

An automated education is a contradiction in terms.

Yeah, I’m going to write about Sebastian Thrun’s pivot again. Why? Because every time I look at that story, I find something else worth writing about in it. In fact, with additional perspective, I’ve come to believe the most important part of that whole article is his use of the word “profound” here:

“I’d aspired to give people a profound education–to teach them something substantial,” Professor Sebastian Thrun tells me when I visit his company, Udacity, in its Mountain View, California, headquarters this past October. “But the data was at odds with this idea.”…

“We were on the front pages of newspapers and magazines, and at the same time, I was realizing, we don’t educate people as others wished, or as I wished. We have a lousy product,” Thrun tells me.

The question then becomes what made Udacity’s courses a “lousy product?” Why couldn’t their courses be “profound?” I’d argue that it’s the lack of the human element. Appointing untrained “mentors” couldn’t get SJSU kids through introductory math. Likewise, turning math into a solo game or giving kids iPads on school trips won’t make a difference either because computers can’t give anyone a profound education. Only people can. Thrun wanted to create teaching machines, but it turns out his machines can’t do what they’re supposed to do with respect to the people who need a profound educational experience the most.

To be fair, this problem goes well beyond MOOCs. Education is an inherently labor-intensive process, which explains why everyone who’s trying to automate it will eventually have to come to the same conclusion that Thrun did. For the sake of variety, consider another subject that I’ve been meaning to get back to for a long time now: automated essay grading.

A while back, Elijah Mayfield of Lightside Labs, made a couple of really interesting appearances over at the tech blog e-Literate. Lightside Labs is working on using computers to assess essays. Notice my change of word there? They specifically state their goal is to help teachers deal with student writing in large volumes, not to do that job for them. Elijah was also incredibly clear that he wants to do this to make assigning writing more rather than less feasible. In short, these people are a lot more teacher-friendly than Sebastian Thrun is.

In order to make these noble intentions feasible, Lightside Labs needs actual teacher-graded essays in order to train the computer to recognize patterns. Do away with teachers and the whole program falls apart. Yet even with teachers playing a huge role in the machine-grading process, there are gigantic holes in what their program can grade well:

Longer reports, like a 10-page term paper, are probably not a good fit for automated assessment. When you start adding section headings, writing becomes less about good style and content, and more about the organization of a document and the flow of information. These aren’t a perfect fit for LightSide’s strengths.

LightSide isn’t going to check grammar. In fact, we don’t have any modules built in that test students’ pluralization, subject-verb agreement, or other textbook rules. If teachers give poor scores to writing that breaks particular rules, then LightSide will learn to do the same. Fundamentally, though, we don’t believe it’s our place to choose what rubric to grade on and what rules to prescribe. That should be up to the teacher; our intelligent software will learn from teacher grades.

Finally, remember that LightSide doesn’t fact-check. Our software learns to spot vocabulary and content that looks like high-quality answers. Sometimes, students will write coherent, well-formed, and on-topic essays that make inaccurate claims about the source material. Usually, our algorithms won’t be intelligent enough to spot well-written but untrue claims. It will, however, grade their writing quality, which is what we aim for.

What’s left isn’t exactly a profound educational experience, is it? I’d argue that’s it’s pretty much rote learning in order to please a computer, rather than the living, breathing person who assigns you a final grade. More importantly, the inability to fact check pretty much disqualifies their system from use in a history class right there. To be fair, I don’t think the Lightside Labs program is aimed at my discipline, but imagine a computer grading program that could be. Imagine that someone had created a program that can tap into all the published sources on the Internet so that it can tell that the War of 1812 actually began in 1812. Is that going to make a profound educational experience possible?

Of course not. History is no more about learning facts than baseball is about learning to hit the ball. History is full of countless subtleties and intricacies that defy immediate understanding. More importantly, there is a human element to both history and baseball that defies easy description. If college, as Thrun suggests, is really all about obtaining eventual employment, then grading by computer is a huge step backwards because no computer will ever be the boss of you. When your boss is a human being, they bring all the foibles that human beings bring to any position of power. Learning to follow the wishes of your professors, even the arbitrary ones, may be the best on the job training that you’ll ever have.

If I’m wrong and the computer will end up being your boss, then I’m afraid we’re all screwed already. Not only will all the cost savings associated with automation flow to the owners of capital, the quality of complicated services that computers provide will become less effective almost by definition. Here’s Nick Carr writing in the Atlantic:

Because automation alters how we act, how we learn, and what we know, it has an ethical dimension. The choices we make, or fail to make, about which tasks we hand off to machines shape our lives and the place we make for ourselves in the world. That has always been true, but in recent years, as the locus of labor-saving technology has shifted from machinery to software, automation has become ever more pervasive, even as its workings have become more hidden from us. Seeking convenience, speed, and efficiency, we rush to off-load work to computers without reflecting on what we might be sacrificing as a result.

That whole article is well worth the read in order to understand the implications of automation in many areas of modern life. It’s not really a Hal 9000 argument at all. Carr’s point seems to be that we’ll forget how to do things, or because these actions will become so systematic, only do them badly if we rely too much on automation to get essential jobs done. While the immediate effects of an automated education may not be plane crashes, to me there would still be an inevitable, obvious drop off in quality.

Last year, I proposed a Turing Test for judging the effectiveness of online education. If a student can’t tell whether they’re being taught by a computer or a person, then the computer is doing a teacher’s job as effectively as a teacher can. But I still don’t think we can ever reach that point.

Machines can teach you rules, but they can never teach you how to break them or especially when breaking the rules is the appropriate response to a particular situation. No wonder Sebastian Thrun wants to do corporate training now. The people most willing to paying for his services are perhaps the only people in society who want education to produce yes men who will never color outside the lines. Call that what you want, but it certainly isn’t a profound educational experience. No matter how powerful our future robot overlords eventually become, only other well-trained human beings can provide that.

The now obligatory post about writing for free.

Last summer, I got an e-mail from Jeff Selingo of the Chronicle. They had started organizing this new project called Chronicle Vitae and they wanted to know if I would be one of the contributors. While I was pleasantly surprised that the Chronicle was interested in featuring the writing of somebody with my politics, I didn’t exactly jump at the chance. You see, I know some people of my political persuasion who’ve had bad experiences writing regularly for the Chronicle. Besides that, I wanted to know exactly what I was committing to and what exactly I’d be getting in return.

Yes, I was rude enough to ask the Chronicle about money. I did this well before getting paid for writing became all the rage because I don’t really want to write for free anymore. Yet I do anyway. Does this make me a bad person? Am I putting professional journalists out of work? Am I contributing to a system of naked exploitation?

I agree with what Derek Thompson wrote at the Atlantic: “It’s complicated.” While I’m not sure this is at all original, here’s my explanation of how I sort it out in my own mind:

Perhaps the greatest thing about having tenure is that I can write what I want for whomever I want to now. I can’t tell you how disheartening it is to spend seven years on a dissertation, five years on revision and have the final product sell a whopping total of 400 copies worldwide. This is not writing for a living or even the pittance of a living. It’s writing for tenure, and there are plenty of worthy presses out there who are more than willing to help you achieve that end – even if you have to buy 25% of those 400 copies yourself so that the press can at least break even.

Honestly, I’m beginning to feel the same way about academic journals. I’ve done my fair share of articles in my time, but I’ve never gotten even one ounce of feedback or encouragement from anybody who has ever read them after publication. Perhaps that’s because none of my articles have been any good, but I can’t shake the sneaking suspicion that it’s actually because almost nobody has ever read them. I spent five years [FIVE YEARS!!!] going back and forth with Technology and Culture to get this article published. While I love the result dearly, I have no idea why I bothered anymore. And, of course, I never made a dime off of it. But then again I didn’t expect to either.

Blogging has the decided advantage of being a lot more fun than writing for purely academic audiences. When nobody read this blog, I told myself that I was doing it for therapy. Now that people do read this blog, I tell myself that I’m doing it for my twin causes: faculty rights and faculty prerogatives for faculty at all levels of employment. To turn down the chance to bring those causes to an audience of professors and graduate students of all kinds would have been idiocy on my part.

Besides, as Jeff explained it to me, I actually like the idea. Chronicle Vitae is kind of like the academic LinkedIn, except academics won’t be all confused about why they joined up in the first place. It’s free to access and there’s even a place where grad students and young scholars can sign up for mentoring. What’s not to like? Besides, since I’ll eventually get around to plugging my book there it’s not exactly “free” labor in the Gary Becker sense of that word.

My first post for Chronicle Vitae is up now. You can find links to my future contributions in this space or just join up yourself and follow me once you’ve registered. Either way, I hope to see you there.

Peer grading (still) can’t work.

I’ve been trying very hard not to sound smug lately. While the anti-MOOC bandwagon was once somewhat lonely, it now seems that just about everybody hates MOOCs now, the public, faculty, college presidents and even campus chief information officers. When I read stories like this, I try to remember that folks like Aaron Bady and Siva Vaidhyanathan were writing about the stupidity of MOOCs just as early as I was, and they were doing it far more eloquently than me as well.

What I hope has stood out in my coverage of this subject on this blog has been my tendency to get down in the weeds and explain exactly how MOOCs work. This was the product of my actually taking one. That led directly to one angle of MOOC criticism that I don’t think comes up nearly as often as it should: the obvious flaws in peer grading. Again, peer review is one thing – I use that strategy myself sometimes. However, letting students grade each other’s papers remains a fundamental dereliction of duty. After all, they don’t pay us to chat on Twitter all day, do they?

Yet peer grading survives. Indeed, a team of writing instructors at Ohio State seem to think they’ve made a big stride towards solving the problem with this strategy:

One way to improve peer grading in MOOCs could be to let students grade their peers who graded them.

That’s what a team of writing instructors at Ohio State University decided last spring when they were designing a massive open online course on rhetorical composition, known as WExMOOC.

They built a custom peer-grading system designed to assess not only the quality of the essays submitted by their MOOC students, but also the quality of the feedback that other students in the course contributed after reading the essays.

While the obvious question here is, “Who’s going to grade the grades of the graders?” and I love Frank Pasquale’s line about “graders all the way down,” this would help solve one problem I pointed out in IHE back in March:

Comments were anonymous so the hardest part of the evaluative obligation lacked adequate incentive and accountability.

Know that you’re grades are going to get graded and maybe – just maybe – you’ll do a better job.

Unfortunately, this doesn’t solve the even more obvious problem with peer grading: Students in a writing class aren’t qualified to grade writing (Otherwise, they wouldn’t have enrolled, would they?). Here’s me back in March again:

Good grading technique is difficult enough for graduate students to learn. Because of the size of the course I think I can safely assume that many of my fellow MOOC students inevitably had no history background at all, yet the peer grading structure forced them to evaluate whether other students were actually doing history right.

The implicit assumption of any peer grading arrangement is that students with minimal direction can do what humanities professors get paid to do and I think that’s the fatal flaw of these arrangements.

So why does peer grading continue? Follow the money. When you start with the assumption that your writing class has to have tens of thousands of people in it, then you structure your MOOC around being massive rather than how effectively it will teach writing. That’s why teaching a writing-based MOOC will always be a Devil’s bargain, but, unfortunately, so far the only people who pay the price are the students.

“[W]ith this bird everything is settled.”

The remains of the last known Passenger Pigeon at the Cincinnati Zoo.

“[W]ith a real nightingale we can never tell what is going to be sung, but with this bird [a mechanical nightingale] everything is settled. It can be opened and explained, so that people may understand how the waltzes are formed, and why one note follows upon another.”

– from Hans Christian Andersen, “The Nightingale,” 1844.*

I. The Game of Writing.

Last Thursday, the NYT ran an article about recent innovations in mechanical essay grading. You’ve probably read it by now, but you know the gist even if you haven’t. Geeks everywhere want to spare faculty the burden of grading student essays so that we can concentrate on other things. [Technology isn’t just benign, it’s good for everybody!] I always thought that grading essays was the thing that we humanities professors were supposed to concentrate upon, but then again what do I know?

As this blog has become all-MOOCs, all the time, here is the part of the article that I found most interesting:

Two start-ups, Coursera and Udacity, recently founded by Stanford faculty members to create “massive open online courses,” or MOOCs, are also committed to automated assessment systems because of the value of instant feedback.

“It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.

There is so much packed into those two extraordinary paragraphs that I barely know where to start.  When MOOC providers champion “the value of instant feedback,” my first question is “value to whom?” I do a lot of grading of written essays in the course my job, and I can tell you that the reason this process often takes so long is because I couldn’t possibly give instant feedback to students even if I wanted.

Good essay questions are about ideas. The essays students write should be about ideas too.  That means I have to sit and think about the ideas that students write in order to grade those questions. Instant feedback is therefore only a good thing if you think that writing assignments are something to get past rather than an opportunity for learning or, God forbid, reflection.

And then there’s that Koller quotation, one of a long series of quotes by Coursera’s founders that have continually left my jaw scraping the floor.  Suppose I ask my students to explain the historical impact of the New Deal.  What exactly is the “right” answer?  I always tell my students that I don’t grade on the basis of what their argument is, I grade on the basis of how well they defend it.  How is any artificial intelligence going to evaluate the inevitable issues of morality that good historical questions invoke from students?  It won’t, of course, and that should be a problem.

Perhaps more importantly, when students keep revising and resubmitting, who exactly are they trying to please? Programmers?  What do they know about good writing?  What values do they bring to the table?  Objectivity is not neutrality, as Thomas Haskell once explained.  As I write these words, this comment is at the top of the “Reader Picks” section of the comments under that NYT article:

Last year when my daughter was in 7th grade, her teacher started using computer essay grading. She would write her essay at home, using the computer, and would get a score. My daughter loves to write but got frustrated because the computer insited on correcting the grammatical errors of portions of the essay in which she used poetic language. In order to get a higher score, she begrudgingly changed her essay.

In short, computer grading destroys precisely the kind of creative thinking that writing is supposed to encourage.

Oh yeah, machines also aren’t very good at determining the accuracy of facts, which might be a problem in…you know…history courses.

II. Feedback Schmeedback

Reading that Koller quote also made me wonder exactly what kind of feedback students get when their essays are machine-graded.  After all, when I force students to play the game of writing , I make them write drafts.  On those drafts, I leave lots of comments. Those comments, in turn, serve as a guide to help students do better on their final papers.

So what kind of comments do students get back on machine-graded essays? Are they just blundering around in the dark?  That sounds a lot more frustrating than fun.  That NYT article suggests that the machines “provide general feedback, like telling a student whether an answer was on topic or not,” but what does that mean exactly?

In order to answer these questions, I did what any good 21st Century cyber-citizen does, I asked Twitter.  Follow that link through a long series of tweets and there are some excellent responses (to go with the inevitable less-than-140-character wisecracks).  Nevertheless, I still felt the need to dig deeper into this issue.

From what I can tell online, it appears that the big debate in the world of machine-grading is whether the scores that machines spit out match the same scores awarded by human graders.  Nowhere could I find anything about the machines giving comments, let alone comments that might actually prove useful.  It’s all about numbers, as if the quality of any piece of writing could ever be reduced to a single digit and a couple of categorizations.

Almost none of these computer science geniuses seem to understand that humanities disciplines are humanities disciplines because the answers to the kinds of questions we ask don’t have easy answers.  This is from Slate, published last year, discussing the problem of applying this technology to my actual field of expertise:

Compare and contrast the themes and argument found in the Declaration of Independence to those of other U.S. documents of historical and literary significance, such as the Olive Branch Petition.

Brown University computer scientist Eugene Charniak, an expert in artificial intelligence, says it could take another century for computer software to accurately score an essay written in response to a prompt like this one, because it is so difficult for computers to assess whether a piece of writing demonstrates real knowledge across a subject as broad as American history.

This may explain why Coursera offers peer-grading for one set of its courses, and is so enthusiastic about machine-grading essays for some others.  Indeed, doing this work I realized that the machine-grading problem is just about the exact equivalent of the peer grading problem.  They use these strategies because of the economics involved, not because they’re the best things to do for students.  That’s what makes quotes like this (from the same NYT article) so incredibly infuriating:

“One of our focuses is to help kids learn how to think critically,” said Victor Vuchic, a program officer at the Hewlett Foundation. “It’s probably impossible to do that with multiple-choice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time.”

Notice the slight-of-hand involved there?  Computer graders are much better than multiple-choice tests, not human graders.  Maybe they are, but who says those are our only two options?  As Mark B. Brown has argued, the fact that we’re even having this debate is an acknowledgement of permanent austerity.  In order to prevent professors and students alike from getting up in arms about this entire discussion, the MOOC enthusiasts and computer science geniuses that enable them have to redefine what education means.

III.  “[W]ith this bird everything is settled.”

My goal as a teacher is to get students to decide for themselves what they think about history.  Do the proponents of mechanized grading even care about such things?  The kind of feedback that students get on machine-graded essays (or on peer-graded essays for that matter) suggests no.

As Mark Cheathem has strongly suggested elsewhere, machine-graded essays and scare tactics go together like wine and cheese.  “You must automate everything!,” the profiteering vultures tell us, “Otherwise, the country will fall behind!”  [Isn’t it really interesting that this strategy transcends national boundaries?  You’d think that the international professoriate could all just slow down together and keep ourselves employed, but I don’t have my hopes up.]  If you think this argument is effective on seasoned professors who should really understand the concept of source bias better, imagine how effective it would be on undergraduates.

I can just hear the pitch now:  Don’t learn anything about critical thinking.  Critical thinking can actually impede your job prospects.  It’ll be just like The Organization Man all over again, only this time they’ll have studies to back them up:

[D]uring the great IT boom, the returns to cognitive skill rose.  Since then, the process has gone into reverse: demand for cognitive tasks is falling. Perhaps this is because installing robots consumes more resources than maintaining them, or perhaps it’s simply that the robots are doing an increasing number of those cognitive tasks.  But whatever the reason, we no longer want or need so many skilled workers doing non-routine tasks with a big analytical component.  The workers who can’t get those jobs are taking less skilled ones.  The lowest-skilled workers are dropping out entirely, many of them probably ending up on disability.

There are 115,000 janitors with college degrees in the United States.  Therefore, anybody who gets one must be a sucker.  Of course, not having a college degree will pretty much doom your chances of getting one of the remaining jobs that require critical thinking (and its corresponding pay level), but who wants to stand in the way of a newly emerging cliché?

What we do know is that cheapening education this way will assuredly put a lot of humanities professors (especially already-underemployed adjuncts without the protection of tenure) onto the unemployment line.  I say if we fall for these scare tactics and accept the values that mechanized grading represents, then we deserve to be there. Instead, we need to make the case that the skills we teach are important irrespective of how much  money students can earn by using them.  Kind of like listening to the song of a real nightingale.

Certainly, mechanical nightingales have yet to replace real nightingales out in the world. After all, they’re far too expensive. However, the values that the mechanical nightingale represents have done enormous damage to other bird species. Take the Passenger Pigeon, for example. Tens of thousands of those birds used to darken American skies:

Now they’re gone.  I, for one, feel like I’ve missed something, even if looking at a huge flock of birds has no commercial value.

In short, everything about the Passenger Pigeon is now settled – not in the same way that everything about the mechanical nightingale is settled, but settled nonetheless. Devalue critical thinking skills to the point that machines grading essays becomes acceptable and everything about education will be settled as well.  Our students will be settled like the mechanical nightingale is settled, singing the same song every time.  We humanities professors will be settled the same way that the Passenger Pigeon is settled, lucky if someone bothers to stuff us and display us anywhere since we’ll become forgotten relics of a bygone era.

But at least we won’t have to waste our time grading papers.

*  I am, of course, not nearly well-read enough to pick that reference out of thin air.  I got it from one of my favorite books of all time, Rebecca Solnit’s River of Shadows. Also, this post wouldn’t have been possible without the help of a slew of my tweeps, especially Cedar Riener, Mark Cheathem and Rohan Maitzen.

Everyone their own professor (w/ apologies to Carl Becker).

I. Everyone Their Own Sushi Chef.

Last summer, I tried sushi for the first time. I was in Korea at the Noryangjin Fish Market in Seoul and I went into a restaurant with a big banner in English that said, “Tourists welcome.” To that point in time, I had been one of those people who said smarmy things like, “I prefer my dead fish cooked, thank you.” However, having fallen under the spell of Andrew Zimmern [“If it looks good, eat it.”], I thought it was time to give sushi a try.

Some of it was wonderful. I particularly remember the octopus sashimi because it was easily identifiable, and thanks to this scene from “No Reservations,” octopus had become the ultimate in creative Korean dining in my narrow American mind. Inevitably, some of the rest of the sushi I had there tasted awful to me. Unfortunately, the woman who served me there spoke very little English so I had no way to identify which kinds of sushi I liked, and which kinds I didn’t. I lacked guidance.

If I hadn’t had sushi, I never would have rented the film “Jiro Dreams of Sushi” on Netflix last week. That documentary about an 85-year-old sushi master, Jiro Ono, was simultaneously boring and riveting. [My wife lasted with it for only 45 minutes, but later she said to me, “I couldn’t stop thinking about him [meaning Jiro] all day.”] Perhaps the most obvious takeaway from the film for a sushi novice like me is that preparing great sushi is a lot more complicated than it looks.

Towards the end of the movie, a food critic explains that eating at Jiro’s restaurant is a bit like listening to a symphony. The “performance” has movements and is scripted to the last detail. Patrons get one piece a time in a particular order to heighten taste sensations. The pieces that women receive are slightly smaller than those going to men so that their smaller mouths won’t slow down the production. The chefs are trained to see which hand each patron favors so that the pieces can be put on the side of the plate that each person favors too.

Why should anyone care about this? Sure, you can just eat sushi like I did, but don’t you want to make the most out of a new experience? If you understand sushi the way that Jiro does, you can learn more than you ever thought possible. The film (and this is what got my wife thinking so hard) even tells you something really special about the nature of work.

In order to see this subculture in all its glory, you have to have a guide. You have to get a sushi education.

II. Everyone Their Own Librarian.

When I was in graduate school, I used to play a game I called “Stump the Government Docs Librarian.” While I don’t think my dissertation was particularly good, it was well-researched in large part because I managed to find all sorts of extremely obscure reports that weren’t even in the US Government Serial Set thanks to the wonderful help I got at the State Historical Society of Wisconsin.

Obviously, Google Books and other such databases now make finding these kinds of obscure sources rather easy. One click, and all the greatest libraries of the world are at your fingertips. In the future, as these resources become even more powerful, they could actually put libraries and librarians out of business. As Nick Carr recognizes, this has created a certain amount of tension between the good folks putting together the Digital Public Library of America (DPLA) and local institutions:

The DPLA leadership is sensitive to this tension, sometimes to the point of defensiveness. In announcing his appointment, [DPLA Director Dan] Cohen wrote, “The DPLA will in no way replace the thousands of public libraries that are at the heart of so many communities across this country.” Yet the first sentence of the DPLA charter reads, “The Digital Public Library of America (DPLA) will make the cultural and scientific heritage of humanity available, free of charge, to all.” It’s hard to see how the DPLA will be able to fulfill such a broad mission without treading on the turf of local public libraries.

In this environment, librarians will have to provide a different kind of guidance. While we may no longer need them to help us find particular books, librarians can still help researchers figure out how to find needles in a series of gigantic haystacks. For example, what search term should you put in that database?

Besides interlibrary loan, I rely on local librarians to tell me what databases are available to me at our institution. I also need help learning how to improve my searches. As almost any history professor knows, students generally know almost nothing about how to search the web when a Google search of a single word or phrase does not yield usable results. That’s why I include lots of librarian time in every class I teach which requires a research paper. In fact, as the tech has gotten better, I’ve expanded that time rather than cut it back.

III. Everyone Their Own Historian.

In 1931, Carl Becker gave what may be the most famous presidential speech in the history of the American Historical Association. He called it “Everyman His Own Historian.” Sexist language aside, Becker’s speech was a poignant call for historians to recognize that academic history only has a purpose when it meets the needs of the public:

Berate him as we will for not reading our books, Mr. Everyman is stronger than we are, and sooner or later we must adapt our knowledge to his necessities. Otherwise he will leave us to our own devices, leave us it may be to cultivate a species of dry professional arrogance growing out of the thin soil of antiquarian research. Such research, valuable not in itself but for some ulterior purpose, will be of little import except in so far as it is transmuted into common knowledge. The history that lies inert in unread books does no work in the world. The history that does work in the world, the history that influences the course of history, is living history, that pattern of remembered events, whether true or false, that enlarges and enriches the collective specious present, the specious present of Mr. Everyman. It is for this reason that the history of history is a record of the “new history” that in every age rises to confound and supplant the old.

I tend to think of Becker’s speech most often during the periodic “Why can’t we all write like David McCulloch?” dust-ups that periodically echo through my profession. Nevertheless, Becker was no anarchist. He still envisioned a role for professional historians in a world where one did not have to have a Ph.D. in order to write good history:

[The history profession’s] proper function is not to repeat the past but to make use of it, to correct and rationalize for common use Mr. Everyman’s mythological adaptation of what actually happened. We are surely under bond to be as honest and as intelligent as human frailty permits; but the secret of our success in the long run is in conforming to the temper of Mr. Everyman, which we seem to guide only because we are so sure, eventually, to follow it.

We historians, in other words, should provide guidance to people trying to come to grips with their own pasts. After all, nobody has the time to research everything they need to tell their own stories. Professional historians can provide the kind of analysis and perspective that amateur historians cannot or choose not to offer.

Without that perspective and analysis, not even David McCullough could write like David McCullough.

IV. Everyone Their Own Professor.

If nothing else, MOOCs [You just knew I’d get to them eventually, didn’t you?] have brought the “Everyman His Own Historian” problem to every discipline in Academia. After all, why should I pay to go to college if I can listen to all the best professors in the world do their thing for free? In fact, if I run their lectures on 150% speed, I can learn everything I need to know in less time that it actually took for them to tell their stories in the first place! And I can do it at home in my pajamas! How can that not be progress?

Not so fast MOOC maniacs. Even the author of DIY U has noted that MOOCs aren’t an education by themselves. Take it away, Anya Kamenetz:

But I have something to say about MOOCs. Specifically about the quality of pedagogy in MOOCs as offered by platforms like Coursera and Udacity and edX. David Wiley, who has taught me a lot of stuff, said this at least five years ago, actually. MOOCs are content. Content is infrastructure. Infrastructure is just the first step.

MOOCs are content = a MOOC is not a course.

I suspect she and I would differ greatly on how much guidance a student needs after the MOOC starts, but isn’t it better to have more guidance rather than less? What too many people don’t understand is that the inevitable effect of MOOCs will be to take that guidance away entirely for most students.

This is what makes members of the MOOC Suicide Squad members of the MOOC Suicide Squad. When I read this post by Mark McDayter, I said to myself, “He’s solved the Cathy Davidson problem!,” namely how to deal with an educator whose goals you embrace, but whose methods will make those goals harder to achieve. You explain the political context in which those methods must inescapably operate:

Davidson, I am reasonably confident, does not support the gutting of Humanities departments and the replacement of teaching faculty with MOOCs. Indeed, she explicitly says as much. But her adoption of the language of the techno-enthusiasts is not nearly nuanced or critical enough to avoid giving aid and comfort to The Enemy.

Who is the Enemy? There are people who are enemies of higher education in general and people who are enemies of professors in particular. We will never win over enemies of higher education in general, who are often very conservative people who think that students can learn anything worth learning all by themselves with no access to professors at all. However, the enemies of professors in particular don’t always recognize that they are enemies of professors. Our job is to show them the light of reason.

If we professors can’t explain why the guidance we provide is an essential part of the college experience, we deserve the fate that inevitably awaits us.*

* That last link is to a Chronicle of Higher Ed article that’s subscription only as I write this, but you can still see my point here just by reading the headline.

Real college classes have writing assignments and required reading.

Do most MOOCs have required reading? I’ve been conversing with the proprietor of the blog Capitalist Imperialist Pig about that question in the comments here. They challenged me to look at all the excellent readings in two MOOCs, Gregory Nagy’s Ancient Greek Hero and Dan Ariely’s Coursera MOOC from Duke, so I did.*

Here’s the “Suggested Reading” statement for Ariely’s MOOC:

I will cover some of the material that is in my 3 books Predictably Irrational, Revised and Expanded Edition: The Hidden Forces That Shape Our Decisions (2008), The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home (2010), and The Honest Truth About Dishonesty: How We Lie to Everyone—Especially Ourselves (2012).

In other words, there’s no required reading, and one writing assignment which is (of course) peer-graded.

The Nagy course is better on reading. There’s a lot of free downloadable translations of Greek texts that you clearly need to read in order to pass the typical mulltiple choice lecture quizzes. But here’s the assessment and evaluation portion:

Students will be evaluated on assessment performance and participation. Assessments will be conducted each “hour” of the course. These will consist of quizzes on the reading (names, places, who is speaking to whom, etc.), as well as the application of principles and concepts central to the course.

Class participation, multiple choice quizzes and no writing at all.

Do you see a trend yet? Then consider this:

From #UTAustin #MOOC on 20th c. ideas: "While not formally required, prior to or during the course, you might wish to read the book" #USIH

— LD Burnett (@LDBurnett) March 30, 2013

Yes, that’s right. You can learn all about great 20th Century ideas at the University of Texas without having to read about any of them. And don’t forget about Harvard lite (law school edition)!

The trend should be clear now: MOOC providers don’t want to scare off potential students with too much work. Talk about teaching in a strait jacket! This is exactly why higher education should never be privatized in the first place. It degrades the quality of the product…a lot.

I’m sorry if this bursts anyone’s bubble, but watching videos on the Internet and maybe writing a few very short essays that the professor never sees isn’t college. Real college classes have writing assignments and required reading. Real college classes require access to the professor. To say MOOCs like these can somehow replace an actual college education is tantamount to fraud.

But that fraud isn’t just going to hurt students left with no access to college but for MOOCs. It’s going to hurt society in general too. This is from UW-Madison History Professor (Go Badgers!) Bill Cronon’s presidential address to the last American Historical Association convention (no subscription needed). [Warning: this story will make any real English or History teacher (and probably more than a few others) cry.]:

Still more poignant and worrisome was the young man who came up to me after a lecture I had just given at another university introducing the major themes of the very long book about Portage, Wisconsin, on which I have been working for longer than I care to admit. I sometimes describe that book as “Michener-length,” though that is a reference few students born in the past thirty years would recognize. So I usually add that I expect the final book to be at least five or six hundred pages long, covering as it does the history of this small Midwestern town from the glacier to now. The illustrated talk I give about Portage is intended to be a crowd-pleaser, with lots of engaging images and stories, and at the end of this particular lecture, a shy young man came up to say how much he had enjoyed it. I thanked him for his praise, but was then mystified when he added that he was very sorry he would never be able to read the book on which my talk was based. I sheepishly told him that although I was taking a long time to finish it, it would eventually be published, and he would certainly be able to read it then. He shook his head and said that was not what he meant. He reminded me that I had described the book as being more than five hundred pages long. Then, with a sad and embarrassed look on his face, he said he was simply incapable of reading such a book, that he had never in his life read anything so long. I was taken aback, but I am quite certain he was speaking in earnest, and that his regret was quite real.

Educating the masses without fixing the problem which that story represents isn’t really educating the masses, no matter what the MOOC maniacs tell you. At least we can blame the delusions of the venture capitalists on self interest. What’s the excuse of everyone in academia who should actually know better?

* These MOOC syllabus links require accounts from the MOOC providers in question. And yes, if you’re wondering, I did get an edX account in order to write this post. [The things I do for this blog!]

Update: A reliable source tells me that I screwed up in my quick read of Professor Ariely’s syllabus. Apparently, there is a substantial list of regular readings scattered around the Internet that I didn’t see. That would make his class more like Professor Nagy’s, a MOOC with a higher workload than most. My apologies to both Professor Ariely and CIP. In defense of my overall point though, I’ll note that the two Coursera world history courses that I’ve been directly involved with have no required reading at all.

Late Update: Since this post is getting a lot of late attention from people who are likely new visitors here, they might also want to read my follow up post on this subject, Some MOOCs are more inferior than others.

For their next trick, they’ll turn lead into gold.

“Will MOOCs Work for Writing?,” asks Chris Friend in Hybrid Pedagogy. The short answer is “No.” Two people who don’t know how to write cannot teach each other how to write. 200,000 people who do not know how to write cannot teach themselves how to write either. There is no such thing as a magic rubric. Teaching writing is labor intensive by definition.

Much to Friend’s credit he doesn’t exactly say “Yes, MOOCs will work for writing.” Instead, he suggests certain aspects of composition MOOCs that can make teaching writing easier anywhere. Unfortunately, the lessons he draws are exactly the kind of lessons that make MOOCs an inferior form of higher education. Due to my now well-known disgust for peer grading, I’ll concentrate on this one:

MOOCs make us rethink or reinforce our conceptions of assessment, pitting the allure and efficiency of mechanized essay grading using machine readers and database-driven plagiarism detection against the more intimate, yet slower, traditional human-scored high-stakes writing. The unsettling spectre of automated essay scoring, applied at large scales, calls us to refine our approaches to writing assessment. Noted composition-assessment scholar Ed White credits the Duke writing MOOC with including reflection, revision, peer review, and challenging reading as pillars of assessment practice, even though the instructor isn’t directly involved in the process. Given most instructors’ distaste for grading, refiguring its role in composition might be the best thing we take from MOOCs.

So just because something more awful than peer-grading exists, we should give up on the idea that every student deserves access to the professor? How cowardly. It’s obvious from the rest of the post that Friend cares deeply about effective education, but the “third way” on MOOCs that he proposes (somewhere between loathing and irrational enthusiasm) is a recipe for disaster.

While some of us debate the possibility of whether the impossible is actually possible, accepted standards are gradually changing so that every aspect of higher ed can be automated, whether or not that’s a good idea (and if you care about quality, it almost always is a bad one). God forbid if we actually start bringing peer grading (as opposed to just peer evaluation) into the physical classroom. Then it will be even easier for the powers that be to eliminate professors entirely as the difference between face-to-face instruction and a MOOC would then actually be minimal.

With all due respect to all the lovely people who are trying to use cutting edge technology to make composition (or other writing-based) MOOCs possible, I don’t need to conduct an experiment to know whether you can turn lead into gold, even if Bill Gates wants it to be so. And while the MOOC enthusiasts continue to fiddle, Rome burns on.

Bang your head (Peer grading edition).

While most people here and elsewhere seem to have appreciated my essay about the futility of peer grading from Inside Higher Ed last week, I have seen enough serious critiques that I want to defend myself here.  While I certainly understand why any superprofessor would want to teach the best MOOCs they can teach, I nonetheless offer what follows as a reality check.

Debbie Morrison, who is open-minded enough to read this blog even though it might well be the polar opposite of hers, rebuts my argument by citing research which suggests the circumstances under which peer grading can be effective:

1) When learners are at a similar skill level.

2) When assignments are low stakes [i.e. when a course is taken for professional development of personal interest…]

3) Where credit is not granted

4) When learners are mature, self-directed and motivated.

5) When learners have experience in learning and navigating within a networked setting [if the review is completed in an open and online setting.]

6) Learners have a developed set of communication skills.

The breakdown in peer grading occurs when the learning environment cannot provide the conditions as mentioned above.

Now that’s all well and good, but the sheer massiveness of a MOOC combined with Coursera’s obligation towards its investors to eventually turn a profit pretty much assures that every single one of those conditions will be violated at one point or another.  More importantly, the rich university administrations that produce course content as well as the poor university administrations that long to replace their faculty with videotaped superprofessors and poorly-paid teaching assistants have every incentive in the world to break every one of those conditions too.

The other critique I particularly appreciated appeared very late in the week at the bottom of the comments to my original article.  Its author signed in as “EnglishTeacher,” and went through the same peer grading process I did as a student in Jeremy Adelman’s course.  They write:

“No one has ever claimed that MOOCs do, can or should replace the full learning experience available to those fortunate enough to be able to be in an engaged on-campus classroom.  And no one has ever claimed that students can take the place of humanities professors.”

Actually, Daphne Koller of Coursera just claimed that students can do a BETTER job at grading essays than humanities professors about a week and a half ago.  With respect to taking over the rest of any particular professor’s job, if the superprofessor provides all the content and student peers do all the grading, what exactly is left?  Not bloody much.  Yes, we can go from desk to desk like my high school math teacher used to do while we worked through our algebra problems, but what kind of wage is that going to get us (particularly as most professors don’t have union representation like so many secondary school teachers do)?

I admire everyone who wants to experiment with new technology to make higher education better for their students.  I really do.  Unfortunately, while those people bang their heads against a wall as part of a futile quest to build a better mousetrap than the one we already have, the powers that be will still be doing their best to make us all technologically unemployed whether robots can do our jobs any better or not.

In the end, the value of peer grading comes down this:  Who can do a better job at grading students essays, peers or professors?  If the answer is “peers,” then why do professors exist at all?*  If the answer is “professors,” then why are so many people wasting their time trying to figure out a way to make the wrong answer right?  As David Golumbia has explained:

MOOCs are being deployed specifically as part of an economic argument whose consequences for liberal arts education are designed to be explosive: they are designed to make liberal arts education emerge as too expensive for us to afford.

Peer grading, like the MOOCs it facilitates, is designed to make the unacceptable acceptable.  It is a strategy created to fit the contours of permanent austerity rather than for the benefit of our students.  So while I agree that pigs look better with lipstick on them, that doesn’t mean the pig becomes any less porcine.

Should anyone choose to keep banging away at the peer grading problem anyway then be my guest.  Just remember that you have been warned, not just about your prospects for ultimate success, but also of the larger political context in which that banging must inevitably occur.

* By the way, I expect the resignation letters of all humanities professors who answer this way to be tendered as soon as they have time to pack up their offices.