I’ve recently completed an analysis of student comments received in Semester 1 2008. While we provide room for students to comment on every question, the overall comments are the only ones subjected to collective analysis.

These questions are:
What aspects of this unit did you find most helpful? (Analysed under “best aspects”)
What are your suggestions for improving this unit? (Analysed under “needs improvement”)

UNE now only has two faculties: the Faculty of the Professions (incorporating the Schools of Education; Health; Rural Medicine; Law; and Business, Economics and Public Policy), and the Faculty of the Arts and Sciences (incorporating the Schools of Arts; Humanities; Behavioural, Cognitive and Social Sciences; Science and Technology; and Environmental and Rural Science).

Results were analysed by faculty, and while many issues were in common, what was most interesting was those that weren’t.

In Semester 1 2008 UNE was forced to move to Blackboard from WebCT, which unfortunately did not go as smoothly as desired and resulted in seemingly unending student complaints throughout the semester. When it came to end of semester unit surveys however, either the students had complained enough and had given up or the semester complainants were a small but vociferous group of students.

The net result was for the Faculty of the Professions there was no dip in ratings, but there were a significant number of student complaints about Blackboard. However for the Faculty of the Arts and Sciences there was a slight dip in ratings, but there were very few student complaints about Blackboard.

Secondly, significant numbers of students in the Faculty of the Professions talked about instrumental issues. For ON campus students (dominantly school leavers), this was reflected in statements along the lines of: this unit was great/poor because we knew exactly/didn’t know at all what would be in the exam, the teacher was great/poor because they explained exactly/didn’t explain how to do the assignment etc. OF campus students held the same attitude but now it was transformed into a focus on structured learning. Their comments were along the lines of: this unit was great/poor because I learnt a lot/didn’t learn much on the topic, the teacher was great/poor because they led an online discussion that really helped me learn/didn’t facilitate or participate in the online discussion, the assignment was too heavily weighted for the amount of work involved, why have an exam when you don’t learn anything in it etc.

When it came to the Faculty of Arts and Sciences I was expecting to see similar comments, but they just weren’t there!

What does this mean? Some people I’ve talked to have suggested: Professions students complain more, education students in particular think they know how education should work, sciences students aren’t going to complain about exams because they always have them.

Personally I have no idea, but it is really interesting. Perhaps an approach to study survey would help shed light on the issue. Any suggestions would be welcome.

Cathryn

Monday 15th September 2008

I’ve just returned from the AES (Australasian Evaluation Society) Conference in Perth. I’m still recovering from a week with poor sleep (the hotel bed wasn’t soft enough) and a Friday night with no sleep on the red-eye coming back, but am managing to function at a reasonable level at work today.

The conference was very interesting as it was all about the larger concept of evaluation. It’s very clear that in Higher Education we tend to mis-use the term ‘evaluation’, with it generally being interchangeable with student feedback surveys. That is, of course, only one piece of information that should be considered with others in an evaluation process.

What’s of more concern is the importance give to student feedback, without consideration of the larger context. Perhaps a unit is scoring badly just because it is based on a different way of learning than most of the others around it. That doesn’t necessarily mean it’s poor - it might in fact be a vital part of broadening a student’s experience. And if compulsory, it would be sure to rate poorly.

In developing a new policy and instrument, the broadening of evaluation was considered, with the Evaluation Policy clearly defining ‘evaluation’ and ’student feedback survey’ to be quite different things. So it’s always been lurking around in my mind, but now it’s come right up front. I’m thinking now about how to build an evaluation framework for units and teaching. Our QM Policy calls for annual monitoring of units based on a range of criteria, but it doesn’t go much deeper. I’m wondering about how do we build a set of criteria that provide a sound basis for this.

Similarly, what sort of framework should support the collection of information for staff appraisal and development? And do we need a portfolio to store and support it? The literature points to a large number of items that can be collected, but are any of these key indicators of performance? And which are important at any particular university?

Love to hear your comments.

Cathryn McCormack
Academic Developer (Evaluation)
UNE

I hope many of you find writing easier than I do. I’ve been struggling with my 4000 word conference paper for the Evaluation Forum in October. It’s another negative presentation - last year I presented on ‘What not to ask in a unit evaluation instrument’, and this year it’s ‘Don’t ask about motivation’ in a universal teaching instrument.

The problem with the question about motivation is that the students who are intrinsically motivated rate teachers low; students who aren’t rate them higher, but it’s only a small proportion who describe teacher characteristics when doing so. The rest talk about the learning materials, the number of students in the class, that they’d rather talk to their tutor than the lecturer etc.

So when considering the results that makes it hard to give them meaning. Does a low score mean the lecturer has a class of intrinsically motivated students (unlikely but possible), was a casual hired for the semester and was given very poor learning materials, had very little time to build a relationship with students because they only lecturered and weren’t a tutor, or are they simply dead boring?

If the answer is meaningless, no point asking the question. Future research - what do we ask instead? I think we can’t ask anything on a universal instrument, any questions have to be more specifically related to the learning and teaching environment. For instance, an individual could ask about how well their lectures were presented, the quality of their tutes, how supported students felt etc. I guess I’ll get somewhere eventually - just not sure where somewhere is!

Cathryn McCormack
Academic Developer (Evaluation)

Hi all,

This blog has been set up for discussion about my research findings in the evaluation of teaching and learning in higher education. It will consider aspects such as: questions, functionality, resources, using results, and reporting back to students.

I’m currently working on the first of these. At UNE we’ve been undertaking a major re-write of our evaluation instruments. The unit (course) instrument we got through quite quickly as it’s reasonably straightforward. The teaching one has been a much greater challenge, and it’s here I’m putting my focus at the moment.

When trying to put together a short set of questions to cover every teaching instance at UNE we had to consider:
- the different teaching roles (unit coordinator, lecturer, tutor, lab demonstrator, etc)
- the different teaching modes (classroom based, online, mixed mode)
- what aspects of teaching students are able to identify
- the ability of students to rate those aspects of teaching

The last two are probably the most difficult, as students differ in their ability to self-reflect, and their ability to attribute responsibility. For instance, if a student participates in a really well facilitated tutorial group, they may just think they were in a really good group this year, rather than the tutor being a brilliant facilitator.

Love to hear what attributes of teaching you think students are able to identify.

Cathryn McCormack
Academic Developer (Evaluation)
Teaching and Learning Centre
University of New England
Tel: (02) 6773 3480
Email: cathryn.mccormack@une.edu.au

Welcome to Blog.une.edu.au. This is your first post. Edit or delete it, then start blogging!