Average is over, higher education, MOOC

Average is over: Future of Academia

In Average is over Tyler Cowan argues that the future of work will be about working with intelligent machines and how effectively we will do that, will depend a lot on education.

Part of the story about education is online education as one of the places where new information technologies is playing a role. Think MOOCs, but also of blogs, Twitter, Wikipedia, YouTube and TED. Cowan argues that there are new ways of learning and they are based on the following principles:

  • Time-shifting,
  • User control,
  • Direct feedback,
  • Online communities, and
  • The packaging of information into much smaller bits than the traditional lecture or texbook chapter.

So does he believe in the “disruption” and “revolution” that people have been blogging about? Not quite. He highlights some of the important features of the economics of online education that will influence the role it can play: that it will be cheap, that it is flexible, that the profits from teaching innovations will be high and that it allows for much more precise measurement of learning. But then goes on to argue that online education and learning games will supplement current systems.

He makes an interesting point about who will provide online education to supplement current systems. The MOOC model is to be replicable and universal. The business model of universities is to market the quality of exclusivity. Though Harvard and Princeton provide MOOCs, they are unlikely to dilute their brands by providing credits and having thousands of partial alums walking around.

Cowan says that online education will be egalitarian in a specific way: smart, motivated learners from non-elite communities will use it to get ahead, but it will not push the uninterested student to the head of the pack. “This kind of learning is driven by the hunger for knowledge, not by a desire to show off your talent or to “signal” as we economists say”.

Given all this, what will the new world of face-to-face teaching look like?

Probably something like the so-called flipped classroom that everyone is talking about: watch a video on YouTube, come to class for some discussion, practice to homework problems on the e-learning system. Cowan writes: “It will become increasingly apparent how much of current education is driven by human weakness, namely the inability of most students to simply sit down and learning something on their own”. He argues that the better performing students will be treated like the chess prodigy’s are today – they use intelligent software to teach themselves, they cooperate with each other and they get guidance and feedback from coaches. “Their conscientiousness, and the understanding that high wages await them in the world, will enforce hard work and discipline”. The lesser performing students will specialise in receiving motivation. In this, there are some choice bits that made me smile:

Education will become more like the Marines, full of discipline and team spirit.

The teacher is first and foremost a role model and a motivator and to some extent an entertainer.

In the longer run, professors will need to become more like motivational coaches and missionaries.

We could think of the forthcoming educational model as professor as impressario.

Let’s treat professors more like athletics coaches, personal therapists, and preachers, because that is what they will evolve to be.

 And then finally:

Of course, educational institutions are not ready to admit how much they share with churches. These temples of secularism don’t want to admit that they are about simple tasks such as motivating the slugs or acculturating people into the work habits and sociological expectations of the so-called educated class.

I think that it is an excellent chapter and well worth the read.

And what do I think it means for me and colleagues in the School? For example, should I jump into edtech and develop an e-guide, make video’s or apps? It depends.

If your resource is uniquely interesting and well explained, you could have a world market for it. If you are just animating your PowerPoints because the Dean says so, you are probably better off curating resources than creating it. I think that there is already a lot of good resources out there. My contribution can be in how I put them together for my students.

Whether you decide to create or curate, the next question you should ask yourself is whether it will help the conscientious students to teach themselves, or whether it is aimed at inspiring and motivating the others. I frequently hear the hope expressed that interactive pdf’s or slick mobile apps will engage this new generation. But I fear that nice graphics and pop-up messages are the sort of things that everyone gets used to very quickly. If we accept the idea of being coaches and preachers I think that it is obvious that you should spend your extra time on your live show in class, on getting to know your student, on being that real-life exemplar – not creating more e-resources.

I think that we should all think back to the professors who inspired us in the days before e-learning. Were they the ones who read from the book, or had perfect over-head projector transparencies, or a nice list of prescribed reading? The modern day equivalents are the e-guides and apps. Or were they the ones who could motivate and inspire? Now put that in your twitter.

dashboards, Economics, higher education, management, scorecards

A few thoughts on management and dashboards

I am coing up to one year in my management job at the end of this semester and often hope that I am doing more good than harm. Last week I found a post that made me think about different approaches to the job.

Chris Blattman wrote a post in response to Bill Gates’ write up of Morten Jerven’s book Poor numbers. Gates wrote that more resources need to be devoted to get basic GDP number right. Blattman responded that everyone would like to see better GDP numbers, but it is not really a top-ten constraint facing poor countries. He sums it up as:

The problem with those of us in the development complex, be we academics or Presidents or foundations or NGOs, is we want the world nicely ordered with levers to pull and a dashboard to monitor. And so we put a lot of energies into levers and dashboards and monitors.

Even if that is the way the world works, there is an opportunity cost and Blattman goes on to outline some of the other types of missing information that we also would  like to know more about.

Which brings me to my point about management and dashboards. I recently completed my own performance agreement. It is in dashboard, scorecard format on the intranet. It breaksdown the university’s plan to a campus plan and “cascades” down from the rector, to the vice-rectors, to the dean’s performance agreement. There are 187 target items have been weighed and will have to be scored at the end of the year.

I can see the appeal of doing it this way, it reduces complexity to lots of nice targets, it can be fine-tuned, the automation will save someone somewhere a bit of time. But I also have doubts about whether this is the way to ensure academic excellence (to the extent that any head of school can do that anyway). I want to throw out the questions to the academics out there:

  • What makes for a good Head of School / Chairperson / Director?
  • What are the things that we need to measure to manage?

I want to paraphrase Blattman: Reducing frictions and eliminating constraints is maybe the best thing outsiders managers can try to help with, freeing entrepreneurs and citizens academics to do their thing.

#higher-ed, higher education, lecturer evaluation

More on lecturer evaluation

In August I wrote a post on lecturer evaluation, having used factor analysis to narrow down the characteristics that our students think make for a good lecturer. The questionnaire has 25 items that students score on a 4-point scale from strongly disagree to strongly agree. The items include things like: The lecturer plans thoroughly and prepares thoroughly for contact sessions. Or, the lecturer makes use of multimedia in support of learning. The lecturer may also explain the relevance of concepts and theories, or explain the relationship between theory and practice. The result was the following:

Component 1 seemed like the baseline characteristics of a good lecturer – being friendly, accessible, fair, offering support, an all-round professional that can explain the work. It accounted for 19% of the variance in the data. Component 2 looked like the things that you have to learn to become a good lecturer. It is not only about knowing your field, you also need to know a bit about formulating outcomes, action verbs, putting it into a study guide and using it. This explained 15% of the variance. Component 3 looked like it was about the enthusiasm and adding value – engaging with the students. And it explained another 15% of the variance. The other 50% of the variance was not explained by these three constructs.

Now, students don’t really like doing these lecturer evaluations for all their courses. After the second or third one they realise that it takes up valuable beer-drinking time. And as with many of these customer satisfaction” type surveys, they don’t really see their inputs making a difference. Unless you are repeating the course next year, you would not know if the lecturer took the low score for providing quick feedback to heart.

On the lecturers’ side, the process does not seem to provide that much useful information or action. If it is going badly in a course, the School director knows about it long before the lecturer evaluation is done. Low scores on a few items might get a mention at the meeting where next year’s teaching load is assigned. If you are really keen, scoring low on “basing assesment on learning outcomes” might get you to enroll in the next assessment training course on campus, but then you also do have some articles to write. At best (and worst) the aggregate lecturer evaluation score matters for the top-20% bonus at the end of the year.

Which brings me to my point, if we are capturing half of what makes for a good lecturer and the result is not particularly useful to students or lecturers, shouldn’t we replace the whole system with a simple vote by text message: rate your lecturer with score out of 100.

This has some support in what has been written about so-called “thin slicing”: In his book Blink, Malcolm Gladwell writes about people’s ability to thin slice – to judge what is good or important from a narrow period of experience. He goes on to argue that having too much information can interfere with the accuracy of a judgement.

To get an idea of the difference between our questionnaire score and a ‘thin-sliced’ score we asked our students for both this semester and the results are as follows. For the analysis I have 869 usable responses from undergraduates in 5 modules for 9 lecturers.

The average score from the questionnaire was 85.7% and the standard deviation 13.3. The figure shows the distribution between the courses.

The average of the thin slice scores out of 100 was 82% with a standard deviation of 17. The correlation coefficient between the two sets of scores was .741 and it was significant at the 5% level. Controlling for the different modules the partial correlation is slightly lower at .731. A paired samplest-test showed that the differences between the means is statistically significant.

It seems that the questionnaire is useful to the extent that it ammeliorates a general dislike of the lecturer. For ‘thin slice scores between 0 and 50%, the average questionnaire score was 63%. Having considered all 25 elements of the evaluation it turns out the that lecturer is not so bad as the initial thin slice score out of 100 indicated. This effect gets smaller at higher scores: For thin slice‘ scores between 71 and 80%, the average questionnaire score was 83%.

Thus, for all its shortcomings, it does seem that the questionnaire serves a purpose. Next we’ll have to try a five-point scale and see if we can measure more accurately! 

higher education, research, teaching

A few links and thoughts for managers

Now that I am School director I’m spending more time in meetings, listening to senior managers’ ideas about the campus plan for 2013 and I’m reading more posts in The Chronicle and Inside Higher Ed. In many ways our NWU, or South African challenges are more universal than we think. Here are three quick perspectives.

Last week Higher Education Research Institute at the University of California at Los Angeles released their survey of faculty (academic staff members’) activities and attitudes. The survey reached more than 23 000 faculty members at four-year colleges and universities in the US. The table below shows activities and time spent:

How Faculty Members Report Spending Time, in Hours per Week, 2010-11

  0 1-4 5-8 9-12 13-16 17-20 21+
Schedule teaching 5.8 15.8 34.6 28.6 9.2 3.7 2.1
Preparing for teaching (including grading) 4.9 11.5 24.4 22.4 13.9 12.1 10.7
Advising and counselling students 4.4 56.7 27.1 7.8 2.2 1.1 0.7
Committee work and meetings 7.6 58.0 23.7 6.8 2.2 1.3 0.4
Other administration 30.6 39.7 13.8 6.6 3.4 2.7 3.2
Research 13.1 30.3 19.2 12.9 6.9 6.0 11.7

The report shows that time spent on teaching is less than the years before, but time spent on research is not notably up. Many faculty members spend small but regular chunks of time each week on committee work, advising, research and other activities. I am keen to compare this to our School where most of the staff would fall in the 5-8 hours of teaching per week bracket. It is clear that the faculty members in the survey do not get close the the NWU’s idea of splitting their time 40:40:20 between teaching, research and community / commercialisation.

Another interesting table shows what is happening in class:

Men, Women and Teaching Methods in Used in All or Most Courses

Method Men, 2001-2 Women, 2001-2 Men, 2010-11 Women, 2010-11
Extensive lecturing 54.6% 34.1% 52.7% 33.8%
Class discussions 68.3% 78.9% 78.3% 88.0%
Cooperative learning 32.6% 55.8% 48.5% 68.8%
Student presentations 30.4% 45.2% 36.9% 53.8%

Faculty members seem to be moving to more student-centered approaches, but the men still seem to prefer more lecturing. This is specifically the case in the so-called STEM fields (Science, Technology, Engineering, Mathematics). This week our campus is hosting a conference on teaching excellence and I am excited to go and have a look at what the top lectures are doing in their courses.

Finally, the report shows that these academics are stressed. The challenge for a manager though is that “teaching load” or “committee work” are not the big stressors. Items like self-imposed high expectations, working with under-prepared students and research or publishing demands rank much higher. Empowering staff to take their teaching or research to the next level will require more than just giving them more time. I think that training is one part, but I also like the idea of nudging. Maybe exposing people to that next level quality will inspire their own efforts.

This brings me to two other interesting posts. Margaret Wente writes in the Globe and Mail that Canadian Universities will have to choose between access and quality. The problem starts with government’s goals of access and mass participation. But:

Academics insist that universities must not become job factories. Their mission is to guide students to cultivate the life of the mind. But there’s a big disconnect between what academics want and what students want. For students, higher education is all about the job. Yet, they get virtually no counselling about what returns they can expect for their investment, or what their job prospects might be.

She goes on to argue that changing this will be difficult and outlines a number of points that sound familiar in SA as well: Big classes, more teaching done by part-time staff (adjuncts), all the incentives for research. Her solution is a bold one:

Give more research money to the most productive researchers, and far less to the rest. Let’s abandon our cherished egalitarianism and fund our leading research universities properly, so they can compete on the world stage. Let’s develop a tier of teaching universities that are excellent at what they do. The result would be smaller classes and lower tuition, a win-win all around.

I also do not think that universities should become job factories, and I disagree with the article’s idea that there should be greater government intervention, but I agree that the system as it is set up, is a danger to undergraduate teaching and in the end post-grads and research will suffer for it as well.

Finally, the Chronicle writes about a Michael Rizzo, a full-time lecturer in economics at the University of Rochester, and argues for two tracks for faculty – teaching, or research. The idea put forward is:

But what if the academic work force were made up primarily of two types of faculty members? One, a small proportion of tenure-track professors—those who earn doctoral degrees, do research, train graduate students, teach advanced seminars, and help administrators run the university. And two, a larger portion of full-time instructors, like Mr. Rizzo, who teach undergraduates, help advise them, keep up with developments in the field by reading and attending conferences, but do no research. Instead of earning Ph.D.’s, like those on the tenure track, instructors could stop with a master’s degree, as many in the adjunct teaching pool already do.

At the NWU this idea is heresy: we recently scrapped the idea of a teaching-learning associate professorship. But then management also has this idea that we can’t have all these part-time lecturers on the  school’s budget. Who is supposed to do the teaching then? Or the other way around, if everyone’s teaching, who’s supposed to do the research then? As an economist I am in favour of the idea of the division of labour and specialisation. If everyone has to do everything, there can be limited gains from trade.

In conclusion, I don’t have answers to these different questions, but I do find it interesting that there are many more people out there, thinking about them.

Collaboration, Communication, Creativity, Critical thinking, dissertations, higher education, student essays, student writing

Writing as assessment

This weekend I helped a colleague to grade some student assignments and it got me to thinking about writing as assessment. As a manager I always say that I would rather help someone move a body in the middle of the night, than to help them with marking. I know that there is quite a discussion online about student writing and I have not read enough of it to write anything sensible, but after ploughing through those “research proposals” I do want to ask some questions.

First-off, I just want to say that I do believe that it is important to have students make their thinking and reasoning visible through writing assignments. This can be short or long essays, research papers or “articles”, whatever you want to call them. It is a way to demonstrate the 4C’s: Creativity, Critical Thinking, Collaboration, and Communication. Being able to write simply and clearly will benefit anyone, everywhere, always.

But how can we go about this in a undergraduate Economics course? There is no shortage of topics, my questions are about practical challenges. The groups are large, ranging from 1200 first years, 450 second years and 180 third year students. Thus writing assignments present a lot of marking. How much individual-specific feedback can you give? If you don’t have more than one essay, what use is the feedback? How do you deal with plagiarism?

Maybe there is someone out there who can share some ideas?

I suppose that having a writing assignment (with all its trouble and flaws) is better than having none. Maybe it is possible to use peer-evaluation for the first essay of the semester and have the lecturer grade the second one? Or you can try and get a bunch of post-grads to help read. Or the lecturers can share the load. Turn-it-in helps a bit.

I would love to hear from other academic economists on this. Are you still giving writing assignments? How do do the grading and feedback? Has anyone tried alternative approaches? Please leave a comment…

good lecturers, higher education, lecturer evaluation, university

On lecturer evaluation

Moving offices and attending meetings means that I am writing fewer posts than I would like to. I really wanted to do one about the stuff I discovered when pack up the old office – notes from my PhD, the handbook for new lecturers 1999 etc. But leaving nostalgia aside, once in a while I do management work that also makes for good blog content. Here’s a re-post of the one I wrote for the School blog on lecturer evaluation:

In his book Blink, Malcolm Gladwell writes about people’s ability to “thin slice’ – to judge what is good or important from a narrow period of experience. He goes on to argue that having too much information can interfere with the accuracy of a judgement.
Putting all that to one side, the NWU-Puk has for some time been asking students what makes for a good lecturer? Formally these are known as lecturers’ evaluation and normally it takes place towards the end of a semester, ideally in all courses. We don’t ask students for their gut feel on who are the good lecturers, we ask them about the characteristics of good lecturers using a questionnaire. Is there a case for thin slicing, or does our analysis approach yield results?
Now, I have been to a faculty council meeting where the questionnaire has been pulled apart and people congratulated on their results in almost the same breath. Some Schools use the “old” ones, Institutional academic support services have developed one and then there is the “new” version. A number of Faculties use their own questionnaires because they are different.
Being Economists, our School decided to bring some rigour to the debate. In the first semester of 2012 we used the e-learning platform to administer the “new” questionnaire and with that data, we now have some clues as to what students think makes for a good lecturer.The questionnaire has 25 items that students score on a 4-point scale from strongly disagree to strongly agree. The items include things like: The lecturer plans thoroughly and prepares thoroughly for contact sessions. Or, the lecturer makes use of multimedia in support of learning. The lecturer may also explain the relevance of concepts and theories, or explain the relationship between theory and practice. So what is it that the high-scoring lecturers do? I have always argued that being well-prepared and enthusiastic is more than half the battle. To see what different constructs make up that elusive “good/great” lecturer we used factor analysis to analyse our data.
We have responses from seven undergraduate courses in Economics, International Trade and Risk Management, ranging from first years through to third years, and a total of 1962 observations. There are 12 lecturers involved.
The KMO measure of sampling adequacy was .975, which shows that the data are suitable for factor analysis and the principle component analysis extracted three factors, explaining 50% of the variation in the construct “good/great” lecturer. After a Varimax rotation the items are grouped as follows:
Component 1
Component 2
Component 3
The lecturer is on time for classes The lecturer makes use of the study guide during contact sessions The lecturer encourages students to attend learning support on campus when their performance is weak
The lecturer makes use of multimedia in support of learning / makes effective use of visual aids The lecturer bases assessments on learning outcomes as stated in the study guide The lecturer encourages students to work together during contact sessions
The lecturer plans thoroughly and prepares thoroughly for contact sessions The lecturer explains how outcomes will be assessed The lecturer encourages students to participate in the class discussions
The lecturer is friendly towards students The lecturer presents study material in an organised manner as set out in the study guide The lecturer encourages students to make a greater effort in their studies
The lecturer offers support and assistance when requested to do so The lecturer states learning outcomes I have to master for every contact session The lecturer presents contact sessions that are valuable learning opportunities for me
The lecturer promotes an atmosphere of mutual respect The lecturer explains the relationship between study units The lecturer explains the relationship between theory and practice
The lecturer uses a level of language that I can understand
The lecturer refers to relevant and recent developments in the subject
The lecturer assesses assignments and projects fairly and transparently

The lecturer gives feedback on tests and assignments within a reasonable time

The lecturer explains the relevance of concepts and theories

Component 1 seems like the baseline characteristics of a good lecturer – being friendly, accessible, fair, offering support, an all-round professional that can explain the work. It accounts for 19% of the variance in the data. Component 2 looks like the things that you have to learn to become a good lecturer. It is not only about knowing your field, you also need to know a bit about formulating outcomes, action verbs, putting it into a study guide and using it. This explains 15% of the variance. Component 3 looks like it is about the enthusiasm and adding value – engaging with the students. And it explains another 15% of the variance. The other 50% of the variance is not explained by these three constructs. What describes the other half of being a great lecturer? I would love to hear how students/ lecturers think we can measure that magic. Or should we just thin slice and ask the students to score the lecturer out of a 100?
Finally, I am glad to say that my colleagues in the School did really well in these evaluations and should take a bow.