Wednesday, November 14, 2012

Thoughts on Our First Panel Discussion

(Post by Mark Lewis, Associate Professor of Computer Science at Trinity University)

On Monday 11/12/2012 we had the first panel discussion. The three faculty members at Trinity who are taking MOOCs this semester formed the panel. Attendance was rather good with most of the desks in the collaborative being filled for the majority of the discussion. Another thing that made the discussion a good one was that not everyone in the room, or even on the panel, was really convinced about the efficacy of MOOCs. This lead to more interesting and thought provoking discussions.

In this post I wanted to put down some follow-up ideas that I had related to this discussion. Of course, I was one of the people on the panel who felt that MOOCs could be effective and were worth watching out for. So this post contains my response to two main issues with MOOCs that were raised during the panel and some elaboration on why I fear MOOCs. I agree that the issues are significant problems with current MOOCs. What I want to point out is that they are also correctable.

Problems with Meet-Ups
One of the obvious shortcomings of a MOOC is the lack of actual physical contact with the professor or other students. While there really isn't much that can be done about contact with the professor (that is after all what makes the MOOCs scale so well), there are solutions when it comes to students. Many students in MOOCs schedule meet-ups with other students so that you can have the discussion that can be very vital to the learning process. For the MOOC I was part of, they even pointed out http://www.meetup.com/ as an option for getting together with other class members and for continuing to build a community after the class had ended.

I actually think that this approach works well in some places. Unfortunately, San Antonio is not currently one of those places. As a city, San Antonio has a sufficient population, but I don't believe it has a sufficient number of active MOOC takers. This is a problem that would be solved naturally if MOOCs were to gain more traction. Were they to really take off so that a decent percentage of people use them both as continuing education as well as post-secondary education, then every reasonably sized city in the US would likely have active meet-up groups that could form around the topics.

I could even see a model where the MOOCs look for "local experts" in the topic of the course in major metro areas. These people could probably be given some small compensation to get them to attend meet-ups regularly. Whether that is economically feasible is unclear, but I see this as pushing forward the trend I expect to see in teaching where people are broken into a few superstar content creators and large numbers of "learning coaches".

Shortcomings of Grading
One of the big discussion points of MOOCs is the grading. My CS course was done wonderfully. Automatic grading of code correctness is easy. The MOOC I was enrolled in went further by also checking style with ScalaCheck and giving points for that. They also integrated their tools very nicely so that it was easy for students to submit and get feedback. However, checking coding assignments is only one part of a complete evaluation for most Computer Science courses. That leads into the objection that was brought up in the discussions.


One of the other faculty was taking a Statistics course and the quizzes were all multiple choice. (Full disclosure, I have to admit that I normally use the term "multiple guess" so that tells you how I feel about that type of evaluation.) The problem that she had with this was the fact that she really cares more about the approach that students take more than the final result. She mentioned how seeing the actual work gives her a better ability to see what students are misunderstanding and to modify her teaching methods in the future to improve learning.

In this case, the desire was to see the work involved in doing math to solve problems. Right now the MOOCs do not include a method for doing this. However, this is something that I feel is just waiting for someone to create the right software. I can imagine a tool that includes a formula editor with additional intelligence and perhaps some notation so that students can indicate where one thing leads to another or what they are doing. It is certainly possible to verify that every step in the process is a valid one and to show that the start point and end points are what they should be. Given this, I don't think it is at all unreasonable to expect automatic grading for the work involved in mathematical derivations.

In the area of Computer Science, I have been thinking a fair bit of thinking about how I could set up automatic evaluation of the different types of questions that I like to use for assessment. Coding is one facet, but I also typically ask students to trace code and to do short answers. I don't yet have an approach for auto-grading short answer questions, but I do have an idea for auto-grading tracing questions which I wrote up here.

In addition, I have quite a few problems where students have to draw the data structures that result from performing various operations. By writing a custom drawing program, which includes only the constructs that students need to be able to draw data structures, it is possible to determine if what a student draws is equivalent to the correct answer and if not, how much it differs. One of my projects for this summer is to write a program that does these things so that I can have students do exercises beyond writing code that are automatically graded.

The advent of MOOCs is inevitably going to spur a lot of people to think about ideas like this for doing automatic evaluation of different types of problems that are significant for particular topics. Many of these will be doable well before we have really good short answer or essay auto-graders.


Fear and Economics
At the end of the panel we got into the question of whether people fear MOOCs and the impact that they might have on liberal arts schools. The room was definitely divided on this topic. I personally feel that MOOCs have to be taken seriously because of the reality of economics. As any economist will tell you, people buy on the margin. The price of a good is based on marginal utility, not absolute utility. For colleges, that means the real question is what benefits does a school provide above and beyond the lower priced competition, or how close can they be in quality to the higher priced competition.

Students receive many different things from a college education, but the reality is that the main thing they are paying for is the signaling to get them a better job. The stats clearly bear out the fact that a college degree is a good investment for most majors. (http://cew.georgetown.edu/whatsitworth/) However, that is largely because it is pretty much the only broadly recognized signaling option currently available. If MOOC certificates or badges were to gain traction as a signaling option, then the marginal cost becomes extremely significant.

The costs of a college education can vary quite a bit, but at many small, liberal arts schools, it looks something like four years + $160,000. The four years are significant because you do have to consider the opportunity cost of spending those years in college. The $160,000 needs to include room and board as well as tuition, books, and other fees. Small schools often have a residency requirement so tuition is only part of the total cost. Of course, students have to have food and shelter regardless of how they choose to educate themselves, so maybe some of that could be discounted, but if the comparison was to living at home with parents, taking MOOCs, and going to lots of meet-ups, the MOOC option really can come out at very close to $0. The flexibility of MOOCs might also allow students to get to the point where they have sufficient signaling in less than four years.

So the question is, is the marginal value of actually going to college worth the cost? As of today, I think that the answer is definitely yes, simply because MOOCs are new and they are not yet a valid signaling option. However, as I posted previously, I know of employers who already understand the value of MOOCs because they are requiring employees to take them. There is also acknowledgement of their signaling ability from the fact that MOOCs are able to work as job placement services today.

The reason I worry about how MOOCs will disrupt higher education is not because I think that MOOCs will be as good or better than traditional higher education. I don't. They can't recreate all the elements. I worry about them because they are priced so low that they don't have to match traditional higher education, they just have to get good enough to be considered a valid signaling mechanism. Once that happens, a reasonable fraction of parents and kids out of High School will pick to try that route instead of putting large sums of money into traditional higher ed. As it stands, many colleges are already struggling, either financially, or with admissions. The extra competition is going to drive those institutions that are in trouble under because they simply won't be able to attract students at the tuition rates that are needed to sustain their budgets.

Sunday, November 11, 2012

ModPo Part Two

Over the past few weeks I've had the opportunity to be involved in several conversations between Trinity faculty, staff and administrators about technology as it relates to Trinity's overall strategic plan.  These were lively big picture/vision talks, of which MOOCs were just a small part, but the funny acronym does seem to be popping up everywhere these days.

A recent Inside Higher Ed article argued that rumors of MOOCs disrupting everything about education as we know it are overblown because a MOOC sells something fundamentally different than the high-touch experience one gets at a place like Trinity.  Is the sky really falling?  I don't know. But just this past Sunday, I opened the New York Times Education Life section to read an article declaring this "The Year of the MOOC."  Most poet friends and colleagues still just look at me funny and even giggle when I mention the word.  They're not sure what I mean, and in truth, it doesn't seem like anyone wholly is.  Part of this may be due to the fact that MOOCs themselves are evolving even as I type this sentence.  What seems clear is that most everybody wants to be a part of this MOOC thing whatever it was/is/will be, but how?

Variations of these what and how questions keep coming up for me as I've moved through completing my MOOC course on Modern and Contemporary Poetry.  (And maybe this comes with the territory of my discipline, as poets are particularly concerned with both the what and how of language usage in a poem.)  As I mentioned in an earlier post, the what (content) of the ModPO course continues to be pretty great.  It  moves fast.  In fact, it sometimes moves too fast as I've occasionally gotten behind and pulled a few late night po-video marathons, which, by the way, is how and when many of my students actually do their learning, whether I like it or not.  I'd like to speak more to this piece of how MOOCs might meet real changes in the way students actually communicate, think, read etc. as many of the questions I feel like I'm generating from taking my MOOC are pedagogical rather than technological, but I'm still deep in the weeds of my thinking on all that.  For now, I'll just touch on two things that my MOOC doesn't do so well.

One consistent concern relates to MOOC evaluation methods.  Mod Po requires four close-reading essays to complete the course.  On my first essay, I received four peer comments.  They were all over the map.  You don't receive grades, just what the site calls "peer evaluation/feedback."  One of my anonymous evaluators wrote simply "I think that this is an example of a hard effort essay, unfortunately I think that the author doesn't get the main idea of the poem."  He, or she, was, quite simply, wrong on that.

But I do think that ModPo shows evidence of trying to make feedback better.  Whether students use it or not, the rubric for evaluating essays is solid and similar to the kind of thing I use in my own classes.   I expect my students to be able to identify some aspect of a poem, as well as connect the dots to what it points to/complicates etc.  In short, they should be able to answer the "so what?" question.  ModPo asks this too, with varying results.  One sample rubric asks "Does the paper say something about how Dickinson's dashes work.  If yes, please remark on how effectively, significantly, and/or interestingly the essay deals with dashes."  This "how effectively, significantly and/or interestingly" wording seems useful, but, as any teacher knows, even if you ask the "so what?" question, you don't always get an answer.  I got a few responses that reached beyond "yes, the writer does this" and "no, the writer doesn't do that," but not many, and that's the rub. While this course demonstrates excellent content and sufficient guidance for how students might engage with content, there is little follow-up or interactivity in terms of how well they eventually do it.  That kind of higher order thinking/guidance is the heart of any humanities course.  

Ideally, discussion boards provide a place for more interactive guidance, and I had hoped to spend more quality time there, but I admit I've been mostly distracted by the hundreds of conversations picked up and dropped off.  There are come useful threads happening.  They just don't seem to build in any way that creates collective inquiry and discovery, which is another big part of what makes a traditional classroom experience so meaningful.  Another argument against MOOC type learning is that professors simply can't interact effectively with their students, but Filreis comments on a surprising number of threads.  He's around, a lot actually.  I honestly don't think he sleeps.  Not surprisingly, the discussion boards are most interesting when he drops by.  This is an example of one of the ways in which my MOOC succeeds by being like a traditional classroom with a teacher steering discussion in fertile directions.

In addition to canned videos, the ModPo course features a series of live webcasts over the course of the semester.  During these sessions, Filreis and the TAs meet in real time at the UPenn Writers House and class members can call in/text/email questions.   You can also show up if you live in the Philly area.  From what I can see on the screen, live attendance is small.  During the one I watched last night, the first caller said he was an English teacher, and that he'd really learned a lot from the seminar style of the video discussions.  Filreis replied, "The hardest part about running a seminar is knowing what question to ask, so that it's open enough, but not so open that the things runs amock and you can't possibly finish the discussion in the time period." I want to make sure I'm asking good questions here. What is missing? seems to be the thread through this post.   Some kinds of learning can't be massively scaled, but how might the flexibility and expansiveness of a MOOC like this inform what we're already doing?  For the next post.