"Evaluation is about teaching. In fact, it is best seen as a natural part of teaching. It is a process by which we, in our roles as teachers, collect information about our work and make judgements about actions that might be taken to improve student learning through changes to curriculum, teaching methods and student assessment" Ramsden and Dodds, 1989 p2
Evaluation is about making things better
It is about making teaching easier and more effective
It is about improving the quality of learning
It helps us modify and improve our teaching and courses, by monitoring those aspects which most effect student learning.
It happens when we:
| Investigate | Records and statistics concerning our courses and subjects
The teaching methods we use The resources available to students |
| Reflect | On what we are doing On why we are doing it
|
| Ask | Others (students, colleagues, employers) for their impressions and opinions |
Evaluation involves:
| Planning | What do we wish to investigate? Who is best placed to comment? How do we elicit their responses? |
| Execution | Asking Questions Collecting data |
| Analysis | What have we found out? What strengths can we capitalise on? What weaknesses need addressing? What more do we need to do to clarify things? |
| Response | If an evaluation is to have integrity we must act on the information we collect. |
This booklet has been produced by the Centre for Learning and Teaching to provide a quick guide to evaluation at UTS. In this guide we will provide an overview of evaluation, its uses and benefits and it's relationship to student learning. It also details the various forms of evaluation services available from the CLT. If after reading it, you wish to know more, or have any enquiries regarding specific evaluation issues, please ring the CLT on 330 2200.
The CLT is the support unit for academic staff at UTS and can provide advice and support concerning most areas of evaluation including individual consultation, seminars and workshops, course and subject evaluation as well as providing a wide selection of teaching resources and publications.
To ourselves, there are a number of reasons to do so, we may be seeking reassurance that we are doing a good job, that our students are learning in a meaningful and useful fashion, that innovations we have introduced are working or that we have learnt from previous years' experience.
To other people, such as tenure committees, promotion panels or selection committees.
No matter how good we are we can always improve and this can have consequences at both a personal and public level.
Interview panels may be impressed by someone who is doing a good job but they are even more impressed if we are seen to be responding to feedback and making our teaching, subjects and courses even better.
Similarly it may be reassuring to find that we are doing a good job, but it is even more reassuring to know we are taking steps to strengthen our performance in those areas where we are weakest.
We all would like to know what's going on in our classrooms and subjects and there's only one way to find out -ASK!!! We of course have a very definite idea of what should be happening or what we hope is happening but how closely do our expectations and perceptions match those of others?
Evaluation is any process which allows us to receive feedback on, make judgements about or in some way analyse any aspect of our educational activities.
Different contexts and different audiences require different forms of evaluation, it is important to consider the purposes for which we are doing the evaluation and how and where we are going to use the results before deciding on which form of evaluation we intend to use.
Wherever possible use more than one source of information. Differing sources and methods complement each other allowing us to develop a more complete picture of what is going on. Triangulating between the various forms of feedback enables us to pinpoint sources of problems more accurately.
It is important to distinguish between these two terms, they are often used interchangeably, but in this guide they are used as follows:
Evaluation is a formative process. It is designed to take a diagnostic look at a set of circumstances, identify possible improvements and modifications and then implement these over a period of time. It is an ongoing process. It is forward looking.
Assessment is a general term used to describe all those activities used in judging performance. Assessment can be summative or formative ie used in a retrospective fashion to judge performance or in a diagnostic fashion to improve performance. We may on occasion be able to extract information from the results of assessments of either ourselves or our students which may be of use in the evaluation process but they are not the same thing
Evaluation is seen as an important tool in ensuring teaching quality in Universities and is one of the Higher Education Performance Indicators used by DEET for this purpose.
Ways of ensuring Teaching Quality at UTS are outlined in the 1994 Report "Good Practice and Strategies for Enhancement". This Report highlights specific provisions for the monitoring and management of learning at UTS and the processes, procedures and mechanisms employed to create a supportive learning environment.
It is about improving things. It is about diagnosing strengths and weaknesses in our teaching, our students, our subjects, courses and curricula, capitalising on the strengths and addressing the weaknesses. It is about increasing our awareness of others' expectations and perceptions of our subjects and courses. It is about actively engaging students in the teaching/learning process. It is about not just collecting information but acting upon it.
What follows is a brief introduction to some possible methods to be used in evaluation, for a more detailed account of these and other methods try
Gibbs, G., Habeshaw, S. and Habeshaw, T.
Ramsden, R. and Dodds, A.
Roe, E. and McDonald, R.
Armstrong, J. and Conrad, L.
or any of the more specialised texts referred to where appropriate.
Contact the relevant people
While most evaluation will be concerned with ourselves and areas we have direct control over, there are number of formal positions of responsibility at UTS with specific responsibility for evaluation in certain areas, any evaluation of these areas should involve these people.
Course co-ordinators or equivalent monitor quality, course pre-requisites, maintain contact with Heads of Schools, offer advice on policy matters such as course and subject development and overview the assessment of the course for which they are responsible.
Subject co-ordinators liaise with other subject and course co-ordinators to ensure integrity across the curriculum and integrity within a course.
Co-ordinating examiners and assessors overview each subject in a course. They review subject material, teaching methods, assessment, match between intent and delivery, examination papers and resolve students' problems and needs.
Advisory Committees and School Teaching Development Committees are also used to assist in monitoring and reviewing curricula.
Identify issues to be addressed
General questions to consider:
Develop a process to gather their views and comments
Different contexts, different audiences, different purposes, different respondents, will require different processes to collect input for evaluations. In all instances the primary focus of evaluation should be improving student learning. We may from time to time wish to evaluate how efficient, economical or enjoyable an educational activity is, but in doing so what we should be asking is how economically, efficiently or enjoyably is it facilitating student learning?
Indicators of students learning. It is very difficult to directly monitor whether learning is occurring in a meaningful fashion for our students, so most evaluation must be directed at determining whether we have established an environment likely to encourage our students to adopt an approach which will allow them to learn in a meaningful fashion.
Fortunately many of the factors needed to establish such an environment have been identified through educational research (Ramsden 1992, Entwistle and Ramsden 1983, Moses 1991) Some of the more important ones are:
Clear Goals and Standards If these are absent, students will focus on the assessment activities as the obvious indicators of what is expected of them and will direct their energies to doing the minimum necessary to pass the assessment rather than risk wasting effort on things which are incorrect or irrelevant.
Workload If students perceive themselves as overworked they will attempt to lighten the load by skimming through the work and again only doing the minimum necessary to pass the assessment.
Appropriate Assessment As has been previously indicated students will usually look to the assessment to indicate to them what they need to do and how well. Even if everything else is designed to promote meaningful learning within the students, if the assessment doesn't reinforce this then the students will take their cue from the assessment and adopt whatever approach to learning will best allow them to pass.
Feedback Receiving comprehensive and understandable feedback helps students perceive learning as an active process, of forming ideas, encountering the flaws in them, revising or replacing them in the light of these problems and then starting again, rather than a passive process of absorbing knowledge.
Availability of Teaching Staff and Other Resources Students encouraged to take an active role in the learning process but denied access to adequate resources to fulfil this role quickly become frustrated and disillusioned.
Choice Students who perceive they have some choice in what they learn and the way they learn it develop a sense of owning the process and a greater degree of commitment than those who feel they are simply being pushed down somebody else's path.
Enthusiasm and Commitment of Teaching Staff Enthusiasm is infectious, if students become as enthusiastic and committed to a subject as we are then the rest is much easier.
Subjects - which aspects do we evaluate?
There is no definitive list of those aspects of our subjects which can or should be evaluated but here are some of the more important ones.
This covers a wide range of possible evaluation techniques, primarily concerned with gathering opinions through a number of processes, from a number of sources, most notably our students, ourselves and our colleagues.
Students are one of our best sources of qualitative input, this can be obtained either through talking to them or getting written responses from them.
Interviewing students individually can be very revealing and helpful but is also very time consuming and can be intimidating for the students involved. It should never be attempted without their full cooperation.
Group discussions have the potential to be just as informative and are considerably easier and quicker to prepare and run.
Informal group discussion such as asking a group "How's the course/subject/lecture going?" can be valuable in helping us to make running alterations to the subjects we are currently teaching but is limited in the depth of response it can elicit and should be kept brief and not overused.
Structured discussion can provided more detailed feedback on a subject. Gibbs (1982) describes a number of possible ways such as Nominal Group Technique (Appendix 3) to structure group discussions to maximise the benefits gained from the process. We may wish to facilitate these discussions ourselves so as to hear first hand what the students have to say, however at other times we may be worried that our presence will prevent students from expressing their feelings fully and frankly. On these occasions we can enlist a colleague who is trusted by both ourselves and our students as an 'honest broker' to facilitate the discussion for us and report the findings in appropriate fashion.
Questionnaires using open ended questions (Gibbs etal 1988) can be used to elicit students' opinions. Entire questionnaires using open ended questions can be specifically designed for individual subjects, or alternatively some open ended questions could and should be included in any questionnaires we use even if the majority of the questions require numerical or graded responses.
Colleagues can help provide feedback, by acting as observers in our classes, by reviewing course materials, assessment exercises and subject or lecture plans.
They can also assist us in self evaluation by acting as our partners in debriefing sessions or act as experts appraising our subjects' contents, aims and objectives, or structure.
Colleagues are not only those people working in our department but people from academic development units, people from other schools, faculties or institutions, even experts from industry).
Self evaluation and reflective analysis are approaches to evaluation which may seem more appropriate for evaluating our teaching but in effect evaluation of subjects and evaluation of teaching overlap to a considerable degree. Particularly if we are responsible for the design or resourcing of subjects or their integration with other subjects within a course. We can reflect upon our subjects either verbally through debriefing sessions with our colleagues or by keeping a written record of our experiences and reflections upon them, such as a reflective journal or teaching portfolio (Gibbs 1989).
This refers to any evaluation process which produces numerical results, such as analysing student pass rates or graduate employability or questionnaires using closed ended questions or statements requiring numerical or graded responses. It is easy to make the mistake of assuming that as they are quantifiable and lend themselves to statistical analysis these methods are in some way superior or more credible, but while these quantitative methods can serve a function in evaluation they often tell you less than qualitative methods. The major advantage of these methods are they are quick and simple to administer but as often happens detail and thoroughness are sacrificed to facilitate this speed and simplicity.
Assessment results are a readily available source of quantitative data. Pass rates, average marks and the spread of marks around that average, can all tell us things about how our students are dealing with our subjects (Rowntree 1987). However we should not place too much emphasis on these, as the assessment is itself part of the subject and discrepancies between how our students perform and how we expected them to perform may reflect problems with the assessment as much as with any other part of the subject.
Questionnaires which invite students to grade their responses (usually 1 - 5 strongly disagree - strongly agree) can also be used to get a quantitative representation of our students' view of our subjects.
The CLT administers a Student Feedback on Subjects System on request. This includes a questionnaire can be customised for individual subjects with a free choice of up to fifty questions (either graded response or open ended) which can be selected from the itembank (appendix 5) or designed specifically for the subject being evaluated. These are printed and mailed to you by CLT. Once returned, the questionnaires are analysed and the results are returned to the subject co-ordinator.
Any enquiries concerning the Student Feedback on Subjects System, should be directed to either Hatayi Tas 95142220 or Keith Trigwell 9514 2196
To collect Student Feedback on your Subjects:
The student feedback results will be returned to you as soon as we have processed the data.
It is important to remember that students can comment validly only on
some aspects of the subjects. Students' feedback is one valuable input into
any review; your own reflections, your peers' candid comments and students
work etc are other even more important sources of feedback about your subjects.
If you would like advice on ways to evaluate your subject or teaching,
please phone Keith Trigwell 9514 2196.
Like students, academic staff can benefit from feedback on their work. It is most important that this is a reflective and honest process.
Teaching - which aspects do we evaluate?
There is no one description of good quality teaching. Every teacher has their own strengths and weaknesses. However there are a number of common skills, techniques and behaviours in teaching which we can identify, evaluate and improve on through training and practice (Lally and Myhill 1994, Moses 1991). Here are some suggestions:
Remember that evaluation is formative, ongoing and should come from a variety of sources. Triangulating evidence means correlating the information and data we have received from different sources. Once we have triangulated, we can judge the validity and reliability of the data from a more informed perspective.
As with evaluating subjects there are three main sources for this feedback
Students we can simply ask students informally what they think of a particular teaching method we have employed, or whether an explanation is clear.
Interviews with students or structured group discussions can give a good indication about what is working well and what is not within our teaching (Gibbs 1982, Ramsden and Dobbs 1989). Issues such as whether to facilitate the discussion ourselves or use another staff member become more pointed when the discussion is about our teaching rather than the subject in general and should be given very serious consideration.
Questionnaires using open ended questions allow students to give their opinions anonymously and therefore, hopefully, honestly but don't allow the flexibility of discussion groups. As always we should choose our questions carefully as there are some aspects of our teaching are valid matters for students to comment on such as:
but others are less so, such as:
Self evaluation is something we all do informally every day. It would be rare to walk out of a classroom without giving some thought to what had just gone on - what went well, what could be done better next time. Unfortunately much of this reflection soon gets lost or forgotten as other matters demand our attention and we find ourselves repeating mistakes and failing to gain the full advantage of this reflective process. One way to avoid this is simply keep a journal in which to write down an appraisal of each class immediately it finishes.
Points to consider are:
A reflective journal of course does not have to be limited to these sorts of pragmatic issues (though they are a good place to start) we may choose to reflect on broader issues as well, such as the nature of learning, what constitutes good teaching practice, our expectations of students and theirs of us. The point is to develop a document which will preserve these thoughts and reflections and act as a resource for analysing and improving our teaching. (Boud et al 1985 and Zuber-Skerrit 1993)
Videotaping lessons for viewing later provide us with a rare opportunity to watch ourselves in action and to witness what is happening in our classes from an entirely different perspective. It does take some time and effort to set up and should be used sparingly but it can occasionally provide striking insights into the way we are teaching. Videotapes can also alert us to any unconscious distracting mannerisms of speech or gesture which we may use.
Peer Input can also be greatly facilitated by the use of videotape. A colleague viewing a tape with us may observe weaknesses and strengths we might miss, and may be able to suggest alternative techniques from their own experiences.
Colleagues can from time to time "sit in" on our lecturers, tutorials or practical classes and give us feedback on how they perceived the class to have gone. Needless to say colleagues should only fill this role if invited and we should only invite people we trust. Even then we should not place too much emphasis on this (or any other) single source of feedback, our colleague's perspective will be different to our own or our students but this does not make it any more (or less) valid. It is possible that colleagues will be distracted by differences between our style of teaching and theirs and they may not pay attention to those areas where we most want feedback, it may be useful to provide them with a checklist of those aspects of our teaching we would like them to observe most carefully.
The important thing is to use multiple sources to verify the feedback we are getting.
All the quantitative methods mentioned with regard to evaluation of our subjects are also applicable to evaluations of our teaching, but with the same cautions also applying. Our students marks may tell us a lot about the relevance of our teaching to the assessment tasks but this is of little help if we cannot relate the assessment tasks to our desired educational outcomes.
The CLT makes available Student Feedback on Teaching System. This includes questionnaires based on fifteen set graded response questions and five open ended questions. Staff can customise these questionnaires by adding up to nine graded response questions chosen from the itembank to address issues of particular concern to them in their own teaching.
For advice regarding the Student Feedback on Teaching System, please contact either
Hatayi Tas on 95142220 or Keith Trigwell 9514 2196.
To obtain Student Feedback on Teaching:
Send your order forms to Hatayi Tas, CLT, City Campus. CLT will generate individualised questionnaires for each class and send you the number of copies you request for your students.
The results are confidential to you, but you may want to use them eventually to document your teaching performance. If you want us to keep an identifiable file of your results, so that you may obtain further copies, please indicate this on your order form.
It is important to remember that students can comment validly only on
some aspects of teaching. Students' feedback is valuable input into any
review, but your own reflection on what you want students to learn and how
you are helping them to learn, and your peers' candid comments are even more
important. If you would like advice on ways to evaluate your teaching or
subject, please phone Keith Trigwell 9514 2196.
|
|
|
What to evaluate? Some suggestions:
All the sources and methods of evaluation outlined in other sections are useful in evaluating our courses but a number of alternative inputs are also available to us (Rowntree 1981).
Graduates and employers, are amongst the most important of these As these are often widely dispersed groups questionnaires are the simplest way of seeking their opinions. Again it is important to choose our questions well as employers who have never experienced our courses and graduates who have been in the workforce for several years may be well placed to comment on the objectives and outcomes we are aiming for, but are less well placed to comment on the processes by which we attempt to achieve these objectives. Current students and recent graduates are probably better placed to comment on the mechanics of our courses.
The Course Experience Questionnaire (CEQ) is a questionnaire which was developed to allow us to monitor students' response to our courses across a range of factors similar to the ones mentioned earlier (pg 8). The CEQ has thirty questions which are divided into five identifiable scales Good Teaching, Clear Goals and Standards, Appropriate Workload, Student Independence, and Appropriate Assessment (with a subscale addressing rote memorisation) (Ramsden 1992). A thirty nine question version exists which includes a series of questions concerned with whether our courses encourage students to develop transferable, broadly applicable (generic) skills, and a number of edited or expanded versions have been produced to suit particular purposes. The CEQ is currently used by a number of organisations both internal and external to the university most notably the Graduate Careers Council of Australia who have used it in conjunction with their Graduate Destinations Survey and DEET who have recommended it as a performance indicator for tertiary institutions.
The CLT makes available a version of the CEQ as a specialised questionnaire within the Feedback on Subjects System (order form page 11). When processing CEQ results the CLT will generate the usual results summary involving histograms etc. for individual questions and a scaling summary showing means and standard deviations for each of the previously mentioned scales, (for a breakdown of the CEQ indicating which questions fit in which scale see appendix 2). As with all of the forms of evaluation described so far in this book the CEQ will be appropriate in some situations but not in others and we should consider carefully the purposes of the evaluation before deciding on which instrument to use. Similarly while the CEQ has a number of appealing features, such as ease of administration, a large and increasing bank of normative data for comparisons in most disciplines and widespread recognition and acceptance, it should never be used as the sole source of feedback for any course, all the previously discussed processes for obtaining feedback on subjects and teaching also provide feedback on the course as a whole, as do a number of other sources some of which are listed below.
Additional sources of data:
Existing Documents:
Supplementary information from more informal sources such as:
Small scale formative evaluations are desirable, manageable and make the full blown task less onerous Real benefits can be seen if documentation and evidence can be focused, purposeful and selective, rather than a random sample of information from several years of course operation. We should give emphasis to validity, reliability and interpretation of the evidence and above all to our response to it. As with all other forms of evaluation, course evaluation is not a final grade but an integral part of an ongoing process of fine tuning and updating.
Interpreting the data
Evaluation data are our raw materials. We should use them to consolidate what is known and to build on for the future. Construct an interpretation using this information, and use it to help plan future activities.
Remember that by conducting the evaluation, we are doing the learning. This learning will give us an insight into the stakeholders' perceptions about the situation which has been evaluated. How will we use this insight? What will we do next?
Developing a plan
Evaluation is effective only when the information gained has been acted upon. We must now decide how best to implement any findings, improvements and recommendations into the evaluated area. These changes should be part of an ongoing modification process. Set realistic objectives for improvements and modifications as well as the time scale required to implement, develop and monitor these changes.
The CLT gives advice and seminars on the use of results in areas such as:
For further information call Jo McKenzie on 330 2198 or Keith Trigwell 9514 2196
As mentioned before we can use the results of our evaluations to monitor our own educational activities or to persuade others of the value of what we are doing. Either way we need to document the results of our evaluations and our responses to them.
A teaching portfolio is a good way of keeping a record of what evaluations we have done what the results of these evaluations have been and how we have interpreted and responded to these results. A teaching portfolio is simply a record of our ongoing teaching activities,
It can contain input from a range of sources, recorded in a variety of styles (Gibbs 1989). If we wish we can submit excerpts from a teaching portfolio in support of an application for promotion or tenure. An example is given on the next page of how a teaching portfolio might be used to document an evaluation and the response to it in a way that would be comprehensible and acceptable to an interview panel.
In 1993 I was given responsibility for the subject Philosophy of Instrument Making 2 (subject code 121234) in the Applied Musicology course. In the previous year there had been feedback from the students indicating that they felt they had no opportunity in this subject to raise their own questions or issues and have them discussed. As a consequence they had difficulty in seeing the relevance of the subject to their wider course.
In an attempt to establish a forum where these discussions could take place I replaced a weekly three hour lecture with a series of seminar/tutorials. To help focus and clarify the outcomes of these discussions I included a tutorial assignment at the end of each semester which would contribute towards the student's final grade. After consultation with the other staff involved in the course I decided this should involve producing brief solutions to a number of short problems and that the students should mark this assignment in class as a peer assessment exercise.
At the end of the year I administered a questionnaire (appendix #) which I had designed specifically to investigate students' reaction to the tutorial process including the peer assessment of the tutorial assignments. To get a more detailed picture of how the students felt about the peer assessment of the assignment I asked a colleague Dr Doremi to run a focus group in which the students could openly discuss what they liked or didn't like about the assignment and any changes they might suggest. The students were also asked to fill out the standard feedback on subjects survey used by the department.
Overall the feedback from all these sources was favourable with the standard departmental questionnaire (appendix $) indicating an increase in the perceived relevance of the course (and a little surprisingly an increased perception that the workload for the subject was appropriate) as these are both aspects of a subject which can help promote student learning, this was considered encouraging. The responses to the purpose designed questionnaire (appendix %) indicated that the majority of students favoured the tutorial arrangement over a number of alternatives (80%) and considered the peer assessment of the tutorial assignments to be both fair (94%) and helpful (76%).
There was evidence from both the purpose designed questionnaire and the focus group that the students felt the results of the peer assessed assignments should be included in their final mark for the subject. However during the focus group some reservations emerged. Primarily a number of students felt that as the questions for the assignments were based on the tutorial discussions they tended to favour the more vocal students, some students felt the weighting for these assignments in the final mark was too high (though most felt they should contribute something to the final mark). Other students felt that ambiguities in the problems and in the marking schemes had made it difficult to grade other students work.
In 1994 I continued with the tutorial structure but introduced a second tutorial assignment each semester which I marked, effectively halving the weighting given to the peer assessed assignments. I also began the year with a practice marking session in which the students marked mock assignments and then discussed and developed appropriate marking schemes. I evaluated the process again using the purpose designed questionnaire and asked Dr Fasolate who runs the third year course to sit in on the marking session.
The questionnaires again indicated a favourable response from the students for both the tutorial sessions and the peer marking. Student responses to the open ended "Any other comments?" question (in full in appendix @) included "....now I have a much better idea of what's expected of me , I feel more confident... " and "Seeing other students work has been a big help now I know what a good answer looks like, what I have to work towards.".
Dr Fasolate said that she agreed with the solutions to the problems but she commented that there were two major areas in which students in third year were troubled by misconceptions they brought with them from second year and she was disappointed that we had missed what she considered an ideal opportunity to address these misconceptions in the peer marking process.
In 1995 I ran the course again using the same format as in 1994 but including two questions in the assignment co-written with Dr Fasolate which she hoped would help students address the misconceptions she found problematic in her third year course. The responses to the purpose designed questionnaire were again favourable. A response borne out by a steady increase over the three years of students' ratings for the subject on the departmental questionnaire and I am pleased to say by a corresponding increase in both the average student marks for the subject and the pass rate.
Conducting evaluations:
Effective evaluation leads to:
Remember:
Armstrong, J. and Conrad, L. (1994) Subject Evaluation: A Resource Book for Improving Learning and Teaching: Griffith Institute for Higher Education, Griffith University, Queensland
Boud, D., Keogh, R., and Walker, D. (1985) Reflection: Turning Experience into Learning. London: Kogan Page.
Entwistle, N. and Ramsden, P. (1983) Understanding Student Learning. London: Croom Helm.
Gibbs, G (1982) Eliciting Student Feedback from Structured Group Sessions. Oxford: Oxford Polytechnic, Educational Methods Unit
Gibbs, G., Habeshaw, S., and Habeshaw, T. (1988) 53 Interesting Ways to Appraise Your Teaching. Bristol: Technical and Education Services
Gibbs, G. (1989) Creating a Teaching Profile. Bristol: Technical and Educational Services
Ramsden, P. and Dodds, A. (1989) Improving Teaching and Courses: A Guide to Evaluation. Parkville: Centre for the Study of Higher Education, The University of Melbourne
Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge
Roe, E. and McDonald, R. (1984) Informed Professional Judgement. St Lucia: University of Queensland Press.
Roe, E. (1987) How to Compile a Teaching Portfolio: FAUSA Guide. Melbourne: Federation of Australian University Staff Associations
Rowntree, D (1981) Developing Courses for Students. Maidenhead: McGraw - Hill
Zuber - Skerritt, O. (1993) Improving Learning and Teaching through Action Learning and Action Research. in Higher Education Research and Development, Vol 12 No 1, pp 45 - 58.
Feedback on Subjects
For each subject you may choose up to 50 items from the itembank (though we suggest no more than 25). The open ended questions about the best aspects of the subject and suggestions for improvement (901 and 902 in the itembank) will be automatically included unless you would like to specify alternatives. Please write the numbers of your chosen items on a photocopy of the order form on page 11 . If you wish to gain feedback on aspects of your subject which are not covered by this itembank, we can help you to word appropriate items.
If you wish to use the sets of items for the course experience, the nursing clinical facilitators', or adult education advisors' questionnaires, please write the three letter code for that questionnaire on a photocopy of the order form in the space provided.
Feedback on Teaching
Your questionnaire will contain the statements 1 - 15 as illustrated on the sample questionnaire page 18 , plus up to nine additional items chosen by you from the itembank. The statements you choose will make up section three of the questionnaire. If you wish to gain feedback on aspects of your teaching which are not covered in this itembank we will help you word appropriate items. Please photocopy a copy of the order form for each class you want evaluated and fill in the numbers of the extra questions you want (if any) in the space provided.
Once completed all order forms should be sent to Hatayi Tas at the CLT no less than two weeks prior to the date you intend to administer the questionnaires.
| 0 | Seven point response scale |
| 1 | Five point response scale |
| 9 | Open ended question |
| 1 | 0 | All things considered, how would you rate this subject? |
| 2 | 0 | All things considered, how would you rate the lectures in this subject? |
| 3 | 0 | All things considered, how would you rate the tutorials in this subject? |
| 4 | 0 | All things considered, how would you rate the teaching of your tutor in this subject? |
| 5 | 0 | All things considered, how would you rate this course? |
| 6 | 0 | All things considered, how would you rate your facilitator as clinical teacher? |
| 7 | 0 | All things considered, how would you rate this clinical practicum as a learning experience? |
| 101 | 1 | The objectives of the subject were clear |
| 102 | 1 | The subject covered what the subject description said it would |
| 103 | 1 | The subject outline helped me to understand the subject requirements |
| 104 | 1 | The subject tried to cover too many topics |
| 105 | 1 | The subject covered topics at appropriate depth |
| 106 | 1 | This subject gave me a good introduction to the field |
| 107 | 1 | The subject was sufficiently intellectually stimulating |
| 108 | 1 | There was a good balance between theoretical and practical concerns |
| 109 | 1 | There was a coherent development of theoretical concepts |
| 110 | 1 | There was sufficient emphasis on communication skills |
| 111 | 1 | The subject covers new/unusual terrain. |
| 112 | 1 | The subject made an important contribution to my major area of study |
| 113 | 1 | The degree of difficulty of this subject was appropriate for this stage of the course |
| 110 | 1 | There was sufficient emphasis on communication skills |
| 114 | 1 | The subject assumed too much previous knowledge |
| 115 | 1 | The pre-requisite subjects provided an adequate background for this subject |
| 116 | 1 | The workload was appropriate for a subject at this level |
| 117 | 1 | The workload was greater than that of other subjects at this level. |
| 118 | 1 | We were generally given enough time to understand the things we had to learn |
| 119 | 1 | There were opportunities for students to choose what they wanted to learn |
| 120 | 1 | There were sufficient opportunities for students to pursue areas of interest in the subject |
| 132 | 1 | Creative work was seen as a priority in this subject |
| 122 | 1 | Students in this subject are free to express their own opinions |
| 123 | 1 | Students in this subject are encouraged to ask questions |
| 124 | 1 | The subject is relevant to the course |
| 125 | 1 | The subject is relevant to my professional practice |
| 129 | 1 | Access to academic staff is acceptable. |
| 130 | 1 | Access to administrative staff is acceptable. |
| 131 | 1 | Reading materials were appropriate. |
| 132 | 1 | Library materials were appropriate. |
| 133 | 1 | Subject outlines were informative. |
| 134 | 1 | Feedback on assessment items was satisfactory. |
| 135 | 1 | Feedback on assessment items was timely. |
| 136 | 1 | Feedback on assessment items helped me improve my understanding. |
| 137 | 1 | The subject content was covered satisfactory in the class time available. |
| 138 | 1 | There was sufficient emphasis on the development of interpersonal skills |
| 150 | 1 | I find this subject difficult to relate to |
| 151 | 1 | I do not have much work or organisational experience. |
| 152 | 1 | The exercises are geared towards students with more work experience. |
| 153 | 1 | This subject is difficult to understand |
| 154 | 1 | I could relate this subject to events in the real world |
| 155 | 1 | Exercises which required me to reflect on my own experiences made the subject more relevant |
| 157 | 1 | The videos explained concepts well |
| 158 | 1 | The role plays helped me consolidate what I was learning |
| 159 | 1 | The workshops were too highly structured |
| 160 | 1 | There was not enough time to explore related areas of interest |
| 161 | 1 | The assessment tasks were appropriate for this stage in my course |
| 162 | 1 | This subject encouraged me to search for information in the journals. |
| 163 | 1 | This subject enhanced my ability in clinical/technical skills. |
| 164 | 1 | This subject helped me to integrate theory into practice. |
| 165 | 1 | This subject consolidated my previous learning. |
| 166 | 1 | The self-directed packages helped me to reinforce and extend my learning. |
| 167 | 1 | This subject has developed my health assessment skills. |
| 168 | 1 | I found the subject overall relevant to my career |
| 201 | 1 | I could see the relationships between the topics covered by each lecturer |
| 202 | 1 | Lectures and tutorials were well integrated |
| 203 | 1 | Lecture and laboratory classes were well integrated |
| 205 | 1 | There were too many lectures and not enough tutorial time |
| 206 | 1 | Tutorials helped me understand the lecture material |
| 207 | 1 | Tutorials helped me to understand the course content |
| 208 | 1 | Tutorials helped me to link theory with practice |
| 209 | 1 | Tutorial problem sheets helped me understand the material |
| 210 | 1 | Tutorials/seminars helped me to clarify my own ideas |
| 211 | 1 | Class discussion was a valuable part of the subject |
| 212 | 1 | Group activities assisted my understanding |
| 213 | 1 | Workshops were a useful learning experience |
| 214 | 1 | The case study was a valuable part of the subject |
| 215 | 1 | The simulations was a valuable part of the subject |
| 216 | 1 | Seminars were a useful learning experience |
| 218 | 1 | Simulations helped me develop my skills |
| 219 | 1 | Skill sessions helped me understand the lecture material |
| 220 | 1 | I could see how the topics in this subject fitted together. |
| 221 | 1 | I often had trouble understanding what the lectures were about. |
| 222 | 1 | I found the feedback I was given was helpful in improving my work. |
| 223 | 1 | I have developed my learning skills. |
| 224 | 1 | I now have a better understanding of the language requirements of a tertiary course. |
| 225 | 1 | I have developed my language skills in listening, reading, writing, speaking. |
| 226 | 1 | Opportunities were made to ask questions |
| 250 | 1 | Practical sessions were well organised |
| 251 | 1 | The necessary materials for practical sessions were always available |
| 252 | 1 | Experiments were clearly related to the content taught in lectures |
| 253 | 1 | The laboratory manual assisted my learning |
| 254 | 1 | It was possible to complete the laboratory exercises without really understanding them |
| 255 | 1 | Writing laboratory reports encouraged me to extend my understanding of the subject |
| 256 | 1 | Laboratory sessions allowed for student investigation of areas of interest |
| 257 | 1 | Experiments encouraged me to think critically |
| 258 | 1 | Laboratory sessions helped me to link theory with practice |
| 259 | 1 | I feel that I learned a lot from the laboratory sessions |
| 260 | 1 | The workbook helped me to learn from the laboratory sessions |
| 261 | 1 | There was enough opportunity to practice new skills |
| 262 | 1 | There was enough opportunity to use the equipment. |
| 263 | 1 | Students were encouraged to be creative in their practical work |
| 264 | 1 | Students were given an appropriate amount of guidance for their level of experience |
| 265 | 1 | The computer-assisted learning materials assisted my understanding of the subject |
| 266 | 1 | There was sufficient access to computing resources |
| 267 | 1 | The computing resources were adequate for this subject |
| 268 | 1 | Support staff assisted me when I needed help |
| 269 | 1 | Learning experiences outside the classroom (eg. field work, clinical sessions) were carefully planned |
| 270 | 1 | The computer programs aided my understanding |
| 271 | 1 | Computer simulations assisted my understanding of the subject |
| 272 | 1 | The time spent in doing computer tasks was worthwhile |
| 273 | 1 | I have developed my ability to solve mathematical problems. |
| 274 | 1 | The group activities/workshops helped my understanding. |
| 275 | 1 | The workbook/printed materials were helpful. |
| 301 | 1 | The teaching staff provided a well-coordinated approach to this subject |
| 302 | 1 | The teaching staff presented material in a way which made it easy to understand |
| 303 | 1 | The teaching staff were approachable |
| 304 | 1 | The teaching staff were responsive to student needs |
| 305 | 1 | The teaching stimulated me to follow up on points which were raised |
| 306 | 1 | The teaching challenged my existing ideas about the subject material |
| 307 | 1 | There were sufficient opportunities to ask questions |
| 308 | 1 | Students' previous experience was acknowledged |
| 309 | 1 | Students were encouraged to learn from each other |
| 310 | 1 | Tutorial class organisation allowed for effective student participation |
| 311 | 1 | The tutor was willing to assist me |
| 312 | 1 | The tutor used class time well |
| 313 | 1 | The lecturer made me feel a valuable member of the class |
| 314 | 1 | The tutor encouraged me to participate in class activities |
| 315 | 1 | The tutor welcomed student feedback on the classes |
| 316 | 1 | The tutor has good rapport with students |
| 317 | 1 | The tutor seemed to know the subject matter well |
| 318 | 1 | The tutor communicated his/her enthusiasm for the subject |
| 319 | 1 | The tutor gave me constructive feedback on my assessed work |
| 320 | 1 | The tutor encouraged students to discuss a range of viewpoints |
| 321 | 1 | The tutor seemed aware of students' level of previous knowledge |
| 322 | 1 | The tutor helped students to link theory with practice |
| 323 | 1 | The tutor helped students to work effectively in groups |
| 324 | 1 | The tutor was able to help students with computing tasks |
| 325 | 1 | Guest lecturers explained concepts clearly |
| 326 | 1 | Guest lecturers tried to make the subject interesting |
| 327 | 1 | Guest lecturers made a valuable contribution to the subject |
| 328 | 1 | I found it beneficial to have specialist lecturers for each topic |
| 329 | 1 | Demonstrators were helpful |
| 330 | 1 | Demonstrators seemed to have a good understanding of the practical work |
| 331 | 1 | Demonstrators challenged me to extend my understanding of the subject |
| 333 | 1 | Teaching staff seem very well prepared |
| 334 | 1 | There is too much focus on teaching and learning of facts |
| 335 | 1 | I could generally access teaching staff when I needed to do so |
| 343 | 1 | The lecturer seems to know the subject matter well. |
| 345 | 1 | The lecturer presents material in an interesting way. |
| 346 | 1 | The lecturer communicates his/her enthusiasm for the subject. |
| 347 | 1 | The lecturer has a good rapport with students. |
| 348 | 1 | The lecturer is open to student opinion. |
| 349 | 1 | The lecturer welcomes student feedback on the classes. |
| 350 | 1 | The lecturer is approachable. |
| 351 | 1 | The lecturer is willing to assist me. |
| 352 | 1 | The lecturer responds to student needs. |
| 353 | 1 | The lecturer acknowledges cultural differences amongst students in the class. |
| 354 | 1 | The lecturer makes good use of examples and illustrations. |
| 355 | 1 | The lecturer structures the material well. |
| 356 | 1 | The lecturer stresses important points. |
| 357 | 1 | The lecturer stimulates me to follow up points that are raised. |
| 358 | 1 | The lecturer stretches my mind. |
| 359 | 1 | The lecturer uses class time well. |
| 360 | 1 | The lecturer points out links to other subjects. |
| 361 | 1 | The lecturer demonstrated how the work should be tackled. |
| 362 | 1 | The lecturer stresses social context and relevance in their teaching. |
| 363 | 1 | The lecturer suggests specific ways students can improve. |
| 364 | 1 | The lecturer uses non-sexist language |
| 365 | 1 | The lecturer responds positively to questions from both female and male students |
| 366 | 1 | The lecturer encourages both female and male students to participate in discussion |
| 367 | 1 | The lecturer challenges students of both sexes to extend their thinking |
| 368 | 1 | Demonstrators and technical assistants are equally helpful to both female and male students |
| 369 | 1 | This subject presents both women and men in professional roles |
| 370 | 1 | Case studies used in this subject avoid the use of sexist or racist stereotypes |
| 371 | 1 | The lecturer tries to provide for students with language difficulties |
| 372 | 1 | The lecturer makes students aware of the English language support services available |
| 373 | 1 | The emphasis on self-evaluation has helped develop my reflective skills. |
| 402 | 1 | Students were given sufficient choice in the way they were assessed |
| 403 | 1 | Students were able to choose assessment tasks which suited their interests |
| 404 | 1 | Assessment criteria were clearly related to the subject goals |
| 405 | 1 | Assessment tasks allowed me to demonstrate what I had learned |
| 406 | 1 | Assessment tasks developed to understanding of the material |
| 407 | 1 | Assessment tasks encouraged me to extend my learning beyond the lecture material |
| 408 | 1 | Assessment tasks were relevant to my professional field |
| 409 | 1 | The weighting of each assessment component was appropriate for the amount of work required |
| 410 | 1 | The teaching staff set high standards for assessed work |
| 411 | 1 | I was given sufficient feedback on my assessed work |
| 412 | 1 | Assessment feedback suggested specific ways students can improve |
| 413 | 1 | Assessment feedback made it clear why particular grades were awarded |
| 414 | 1 | Feedback was clearly related to the assessment criteria |
| 415 | 1 | The grading of work seemed fair |
| 416 | 1 | Marked assignments and other work were handed back promptly |
| 417 | 1 | The method of marking group assignments was fair |
| 418 | 1 | The peer assessment system was fair |
| 419 | 1 | Assignments were a useful learning experience |
| 420 | 1 | Assignments encouraged me to read widely |
| 421 | 1 | Weekly quizzes were useful for monitoring my progress |
| 422 | 1 | The reflective journal helped me to develop my understanding of the field |
| 423 | 1 | The reflective journal helped me to develop self-understanding |
| 424 | 1 | There was too much emphasis on assignment work |
| 425 | 1 | There was too much emphasis on formal examinations |
| 426 | 1 | Too much time was spent in writing laboratory reports |
| 427 | 1 | Too much of our learning time was taken up doing assessment tasks |
| 428 | 1 | To do well in this subject all you really need is a good memory |
| 429 | 1 | It would be possible to pass this subject by just working hard around exam time |
| 430 | 1 | It is possible to do well in this subject by studying only the material presented in lectures |
| 431 | 1 | The group assignments improved my ability to work as part of team |
| 432 | 1 | There were clear guidelines for all assessment tasks. |
| 434 | 1 | The methods of assessment used were appropriate for the subject. |
| 435 | 1 | The assessment process used in this subject helped me to learn the material. |
| 436 | 1 | Group projects helped me think through lecture material. |
| 437 | 1 | Assessment in this course tests understanding, not just the recall of facts. |
| 438 | 1 | I received adequate feedback about my progress in the subject. |
| 440 | 1 | Feedback on the assignments was usually provided ONLY in the form of marks and grades |
| 442 | 1 | It would be possible to get through this course by just working hard around assessment times |
| 443 | 1 | The weighting of each assessment component was appropriate for the workload |
| 444 | 1 | I have had enough opportunity to demonstrate what I have learned in this subject. |
| 445 | 1 | The methods of assessment used were appropriate for this subject. |
| 446 | 1 | Assessment requirements are written in clear, simple language. |
| 501 | 1 | The prescribed textbook was useful |
| 502 | 1 | Required reading material was sufficiently accessible |
| 503 | 1 | The reading list made clear what reading was required |
| 504 | 1 | Required reading was appropriate for the subject |
| 505 | 1 | The case book was useful. |
| 506 | 1 | The workbook helped to reinforce and extend my learning |
| 507 | 1 | Handouts helped me to understand the material. |
| 508 | 1 | The lecturer provided useful course notes |
| 509 | 1 | Study guides assisted my learning |
| 510 | 1 | Audiovisual materials assisted my learning |
| 511 | 1 | The UTS Library has an adequate selection of references for this subject |
| 512 | 1 | Administrative support is of high quality |
| 513 | 1 | Library resources are adequate to support learning |
| 514 | 1 | The videos shown assisted my learning. |
| 519 | 1 | Tutorial sessions helped me understand the lecture material. |
| 520 | 1 | Lecture and tutorial classes were well integrated. |
| 521 | 1 | Required reading material was available. |
| 522 | 1 | Recommended texts point out the social relevance of the subject content. |
| 523 | 1 | The subject is relevant to my present employment. |
| 524 | 1 | The content was clearly presented |
| 525 | 1 | I have developed my study skills. |
| 526 | 1 | The group activities/workshops helped my understanding. |
| 527 | 1 | The science content was taught in a way which helped me to understand it. |
| 528 | 1 | The prescribed textbooks and set readings were useful |
| 601 | 1 | I gained a good understanding of the field |
| 602 | 1 | I gained an overview of the major ways of thinking in this subject |
| 603 | 1 | I was encouraged to draw on my previous experience |
| 604 | 1 | I learned to apply principles from this subject in new situations |
| 605 | 1 | I developed the ability to solve problems in this field |
| 606 | 1 | I learned to make connections between this subject and others |
| 607 | 1 | I gained an understanding of how the different components of this subject fit together. |
| 608 | 1 | I developed new ways of thinking about the content of this subject |
| 609 | 1 | I reconsidered many of my former viewpoints |
| 610 | 1 | I changed my understanding of many aspects of this subject |
| 611 | 1 | I was encouraged to reflect on my attitudes and values |
| 612 | 1 | I was encouraged to learn from experience |
| 613 | 1 | I improved my ability to think critically |
| 614 | 1 | I improved my ability to work independently |
| 615 | 1 | I improved my ability to work as part of a team |
| 616 | 1 | I improved my understanding of the way groups can work effectively |
| 617 | 1 | I extended my ability to work creatively |
| 618 | 1 | I improved my discussion skills |
| 619 | 1 | I improved my written communication skills |
| 620 | 1 | I developed skills needed by professionals in this field |
| 621 | 1 | I learned the relevance of this subject to my future profession |
| 622 | 1 | I received a good introduction to the field |
| 623 | 1 | I learned a lot in this subject |
| 624 | 1 | I learned to feel responsible for my own learning |
| 625 | 1 | This subject has helped me to grow and develop personally |
| 626 | 1 | I found this subject interesting |
| 627 | 1 | My level of interest in this subject area has increased |
| 628 | 1 | I gained insights into this area which could be useful in my intended career |
| 629 | 1 | I would like to do further studies in this area |
| 630 | 1 | Assessment strategies are sufficiently variable to evaluate learning fairly |
| 631 | 1 | The group assignments improved my ability to work as part of a team |
| 632 | 1 | I can see the value of this subject to my degree. |
| 633 | 1 | The subject is relevant to my intended career. |
| 635 | 1 | I generally put a lot of effort into trying to understand things which initially seem difficult. |
| 636 | 1 | I often found myself questioning things that I heard in lectures or read in books. |
| 637 | 1 | In trying to understand new ideas, I often tried to relate them to real life situations. |
| 638 | 1 | To do well in this course all you really need is a good memory. |
| 639 | 1 | Student Learning Centre sessions were a useful learning experience. |
| 640 | 1 | The lecturer made me feel comfortable about asking for help. |
| 654 | 1 | I feel that I am making progress in this area. |
| 630 | 1 | |
| 655 | 1 | I could now competently apply the skills I have learned to my professional area. |
| 656 | 1 | I changed my understanding of many aspects of the subjects I studied |
| 657 | 1 | I learned to apply various theories and principles in practice |
| 658 | 1 | I usually set out to understand thoroughly the meaning of what I was asked in the brief |
| 659 | 1 | Often I found I had heard things in lectures without having a chance to really understand them |
| 660 | 1 | When I tackled a new topic, I often asked myself questions about it which the new information should answer |
| 661 | 1 | I needed to look around the course fairly widely before I was satisfied with my own ideas |
| 662 | 1 | I usually didn't have time to think about the implications of what I had studied |
| 663 | 1 | The best way for me to understand the technical terms introduced in this course was to memorise the definitions |
| 630 | 1 | I tried to relate ideas in this course to other things I knew |
| 665 | 1 | I found it helpful to map out this new course for myself by seeing how the ideas fit together |
| 666 | 1 | I have learned to think critically. |
| 667 | 1 | I have learned to make connections between this subject and others. |
| 668 | 1 | I have learned to discuss. |
| 669 | 1 | I reconsidered many of my former viewpoints. |
| 670 | 1 | This subject introduced me to new subject matter and approaches. |
| 701 | 1 | It was always easy to know the standard of work expected of you |
| 702 | 1 | There were few opportunities to choose the particular areas you wanted to study |
| 703 | 1 | The teaching staff of this course motivated students to do their best work |
| 704 | 1 | The workload was too heavy |
| 705 | 1 | Lecturers frequently gave the impression that they hadn't anything to learn from students |
| 706 | 1 | You usually had a clear idea of where you were going and what was expected of you in this course |
| 707 | 1 | Staff put a lot of time into commenting on students' work |
| 708 | 1 | To do well on this course all you really needed was a good memory |
| 709 | 1 | The course seemed to encourage us to develop our own academic interests as far as possible |
| 710 | 1 | It seemed to me that the syllabus tried to cover too many topics |
| 711 | 1 | Students had a great deal of choice over how they were going to learn in this course |
| 712 | 1 | Staff seemed more interested in testing what we had memorised than what we had understood |
| 713 | 1 | It was often hard to discover what was expected of you in this course |
| 714 | 1 | We were generally given enough time to understand the things we had to learn |
| 715 | 1 | The staff made a real effort to understand difficulties students had with their work |
| 716 | 1 | Students were given a lot of choice in the work they had to do |
| 717 | 1 | Teaching staff normally gave helpful feedback on how you were going |
| 718 | 1 | Our lecturers were extremely good at explaining things to us |
| 719 | 1 | The aims and objectives of this course were not made very clear |
| 720 | 1 | Teaching staff worked hard to make their subjects interesting to students |
| 721 | 1 | Too many staff asked us questions just about facts |
| 722 | 1 | There was a lot of pressure on you as a student |
| 723 | 1 | Feedback on student work was usually provided ONLY in the form of marks and grades |
| 724 | 1 | We often discussed with our lecturers or tutors how we were going to learn in this course |
| 725 | 1 | Staff showed no real interest in what students had to say |
| 726 | 1 | It would have been possible to get through this course just by working hard around exam times |
| 727 | 1 | This course really tried to get the best out of all its students |
| 728 | 1 | There was very little choice in this course in the ways you were assessed |
| 729 | 1 | The staff made it clear from the start what they expected from students |
| 730 | 1 | The sheer volume of work to be got through in this course meant you couldn't comprehend it all thoroughly |
| 733 | 1 | This course was pitched at an appropriate level |
| 734 | 1 | This course maintained and enhanced my interest in the field of study |
| 738 | 1 | Studying in this course is having a positive impact on my career |
| 739 | 1 | Technological resources are available to support learning |
| 740 | 1 | There is a coherent development of theoretical concepts throughout this course |
| 741 | 1 | The demands of the course place excessive pressure on students |
| 742 | 1 | Teaching staff are very accessible |
| 743 | 1 | Administrative support is of high quality |
| 744 | 1 | The course was too difficult. |
| 745 | 1 | The course was too long. |
| 750 | 1 | The teaching staff on this course motivated me to do my best work |
| 751 | 1 | The course developed my problem solving skill |
| 752 | 1 | The course sharpened my analytic skills |
| 753 | 1 | The course helped me to develop my ability to work as a team member |
| 754 | 1 | As a result of my course, I feel confident about tackling unfamiliar problems |
| 755 | 1 | The course improved my skills in written communication |
| 756 | 1 | The course was overly theoretical and abstract |
| 757 | 1 | The course helped me to develop the ability to plan my own work |
| 758 | 1 | Overall, I was satisfied with the quality of this course |
| 759 | 1 | There was a good balance between the laboratory and lecture/tutorial component of the course |
| 760 | 1 | Writing laboratory reports often encouraged me to extend my understanding of a subject in the course |
| 761 | 1 | The course helped me to develop confidence in carrying out technical and practical tasks |
| 762 | 1 | The early stages of the course were a good preparation for the industrial training component of the course |
| 763 | 1 | The course has lived up to my expectations. |
| 764 | 1 | The extent to which equity issues are addressed in the course is satisfactory. |
| 765 | 1 | The course is consistent with information I received prior to commencing the course. |
| 766 | 1 | The physical facilities available in this course are adequate. |
| 767 | 1 | The balance between subjects in Stage 1 was Appropriate. |
| 768 | 1 | The course covered what the course description said it would. |
| 769 | 1 | I could see the relationship between different subjects within the course. |
| 770 | 1 | I would recommend this course to other people. |
| 771 | 1 | Teaching staff are very approachable |
| 772 | 1 | Library resources are adequate to support learning |
| 773 | 1 | The case assignments assisted my learning. |
| 774 | 1 | The Academic papers assisted my learning. |
| 775 | 1 | The subjects in this course relate well to one another. |
| 776 | 1 | Support staff (non-academic) were generally very helpful. |
| 778 | 1 | I found the course interesting |
| 779 | 1 | Guest lecturers made a valuable contribution to the course |
| 801 | 1 | The facilitator made sure that the clinical experience was organised in advance with nursing staff |
| 802 | 1 | The facilitator made sure that the clinical experience was negotiated with clients |
| 803 | 1 | The facilitator discussed with me when she/he would be available to give assistance |
| 804 | 1 | The facilitator was aware of my level of previous learning and competence |
| 805 | 1 | The facilitator discussed my learning needs with me |
| 806 | 1 | The facilitator gave me a clear idea of what was expected of me on clinical practicum |
| 807 | 1 | The facilitator allowed me enough opportunity for independent practice |
| 808 | 1 | The facilitator encouraged me to feel responsible for my own learning |
| 809 | 1 | The facilitator provided an appropriate amount of support for my level of experience |
| 810 | 1 | The facilitator negotiated with me about when to intervene to maintain client safety and comfort |
| 811 | 1 | The facilitator made me link theory with clinical practice |
| 812 | 1 | The facilitator made me aware of the legal implications of treatment decisions |
| 813 | 1 | The facilitator prompted me to consider a range of alternative methods of client care |
| 814 | 1 | The facilitator made me aware of aspects of the clinical situation which extended my existing knowledge |
| 815 | 1 | The facilitator motivated me to reflect on my clinical learning |
| 816 | 1 | The facilitator was able to show me how she/he makes decisions about client care |
| 817 | 1 | The facilitator gave me enough feedback on how I was going |
| 818 | 1 | The facilitator gave me feedback which helped me to improve |
| 819 | 1 | The facilitator's feedback was honest |
| 820 | 1 | The facilitator provided feedback at appropriate times, during and/or after the clinical experience |
| 821 | 1 | The facilitator showed interest in my learning |
| 822 | 1 | The facilitator was open to the opinions of others |
| 823 | 1 | The facilitator encouraged students to gain the maximum benefit from sharing learning experiences |
| 824 | 1 | The facilitator seemed to be confident in her/his role as a clinical teacher |
| 825 | 1 | The facilitator was approachable |
| 826 | 1 | The facilitator was sufficiently accessible to me during the clinical practicum |
| 827 | 1 | All things considered, how would you rate your facilitator as a clinical teacher? |
| 828 | 1 | All things considered, how would you rate this clinical practical as a learning |
| 830 | 1 | My advisor seemed to have a good understanding of the course requirements |
| 831 | 1 | My advisor seemed to have a thorough knowledge of the course content |
| 832 | 1 | My advisor seemed aware of my level of previous learning and competence |
| 833 | 1 | My advisor seemed to understand the context in which I was working |
| 834 | 1 | My advisor discussed my learning needs with me |
| 835 | 1 | My advisor prompted me to consider alternative ways to meet my learning needs |
| 836 | 1 | My advisor assisted me to develop my learning contract(s) |
| 837 | 1 | My advisor had a good overview of how I was progressing in the different areas of the course |
| 838 | 1 | My advisor assisted me to extend my learning into new areas |
| 839 | 1 | My advisor was able to suggest appropriate resources to assist me with my learning |
| 840 | 1 | My advisor encouraged me to feel responsible for my own learning |
| 841 | 1 | My advisor was able to challenge my existing attitudes in a non-threatening way |
| 842 | 1 | My advisor was able to challenge me to develop new ideas |
| 843 | 1 | My advisor was able to show me how she/he makes decisions in the professional field |
| 844 | 1 | My advisor helped me to link theory with practice |
| 845 | 1 | My advisor provided an appropriate amount of guidance for my level of experience |
| 846 | 1 | My advisor motivated me to reflect on my learning experiences |
| 847 | 1 | My advisor was able to discuss my journal with me in a helpful way |
| 848 | 1 | My advisor gave me sufficient feedback on my assessed work |
| 849 | 1 | My advisor gave me feedback which helped me to improve |
| 850 | 1 | My advisor was available to see me on a regular basis |
| 851 | 1 | My advisor was sufficiently accessible when I needed assistance |
| 852 | 1 | My advisor made me feel comfortable about asking for advice or assistance |
| 853 | 1 | My advisor showed interest in my learning |
| 854 | 1 | My advisor helped me to grow and develop as a learner |
| 855 | 1 | My advisor seemed confident in her/his advising role |
| 856 | 1 | My advisor encouraged me to put my best effort into the course |
| 857 | 1 | My advisor helped me to get the most out of the course |
| 901 | 9 | What were the best aspects of this subject for you? |
| 902 | 9 | What improvements to the subject would you suggest? |
| 903 | 9 | Please feel free to make any other comments |
| 904 | 9 | Please give your opinions of the assessment in this subject |
| 905 | 9 | What improvements would you suggest for the assessment in this subject? |
| 906 | 9 | How did the nature and/or availability of resources (eg library materials, computing facilities) affect your learning in this subject? |
| 907 | 9 | What are your tutor's strengths in teaching |
| 908 | 9 | What improvements to your tutor's teaching would you suggest |
| 909 | 9 | What was this subject about? (One way to answer this would be to imagine you were describing the subject to a friend.) |
| 910 | 9 | Tutorials would really help me to learn in this subject if they included... |
| 911 | 9 | What would be the characteristics of a good textbook for this subject? |
| 913 | 9 | Which aspects of the subject do you think were most valuable? Why? |
| 914 | 9 | What were the good points about this course? |
| 915 | 9 | Outline what you see to be the strengths of the course to-date. |
| 916 | 9 | Outline what you see to be the weaknesses that need addressing in the course. |
| 924 | 9 | What were your facilitator's strengths as a clinical teacher? |
| 925 | 9 | What improvements could you suggest? |
| 926 | 9 | What effects did the facilitator have on your learning during the clinical practicum? |
| 928 | 9 | Please feel free to make any other comments, particularly in relation to your responses to the previous questions. |
| 929 | 9 | How would you know whether you had learned something from this subject? |
| 930 | 9 | Please list any areas/material that you think you should have learned about during this course but did not. |
| 931 | 9 | Please describe what you think the course was trying to achieve and comment on whether this fitted with your goals and expectations. |
| 940 | 9 | On the back of the answer sheet draw something (a picture, sketch, diagram, doodle etc) which represents your experience of this subject. Add a sentence to explain your drawing. |
| 945 | 9 | What were the best aspects of this course for you? |
| 946 | 9 | Would you be interested in short courses to update your knowledge in particular areas? If yes, which areas? |
| 947 | 9 | What improvements to the course would you suggest? |
| 948 | 9 | What did you like about the osca? |
| 949 | 9 | What improvements would you suggest to the osca itself? |
| 950 | 9 | Do you feel the course enhanced your ability to work with patients/clients, families, self, colleagues? |
| 701 | 1 | It was always easy to know the standard of work expected of you |
| 706 | 1 | You usually had a clear idea of where you were going and what was expected of you in this course |
| 713 | 1 | It was often hard to discover what was expected of you in this course |
| 719 | 1 | The aims and objectives of this course were not made very clear |
| 729 | 1 | The staff made it clear from the start what they expected from students |
| 703 | 1 | The teaching staff of this course motivated students to do their best work |
| 707 | 1 | Staff put a lot of time into commenting on students' work |
| 715 | 1 | The staff made a real effort to understand difficulties students had with their work |
| 717 | 1 | Teaching staff normally gave helpful feedback on how you were going |
| 718 | 1 | Our lecturers were extremely good at explaining things to us |
| 720 | 1 | Teaching staff worked hard to make their subjects interesting to students |
| 727 | 1 | This course really tried to get the best out of all its students |
| 725 | 1 | Staff showed no real interest in what students had to say |
| 704 | 1 | The workload was too heavy |
| 710 | 1 | It seemed to me that the syllabus tried to cover too many topics |
| 714 | 1 | We were generally given enough time to understand the things we had to learn |
| 722 | 1 | There was a lot of pressure on you as a student |
| 730 | 1 | The sheer volume of work to be got through in this course meant you couldn't comprehend it all thoroughly |
| 705 | 1 | Lecturers frequently gave the impression that they hadn't anything to learn from students |
| 723 | 1 | Feedback on student work was usually provided ONLY in the form of marks and grades |
| 726 | 1 | It would have been possible to get through this course just by working hard around exam times |
| 708 | 1 | To do well on this course all you really needed was a good memory |
| 712 | 1 | Staff seemed more interested in testing what we had memorised than what we had understood |
| 721 | 1 | Too many staff asked us questions just about facts |
| 702 | 1 | There were few opportunities to choose the particular areas you wanted to study |
| 709 | 1 | The course seemed to encourage us to develop our own academic interests as far as possible |
| 711 | 1 | Students had a great deal of choice over how they were going to learn in this course |
| 716 | 1 | Students were given a lot of choice in the work they had to do |
| 724 | 1 | We often discussed with our lecturers or tutors how we were going to learn in this course |
| 728 | 1 | There was very little choice in this course in the ways you were assessed |
Nominal Group Techniques are so called because at various points in the process the members of the group are acting as individuals and therefore the group is a group in name only.
Nominal Groups usually work in a number of steps
In late 1994 UTS adopted a teaching development and quality assurance scheme to assist in the improvement of the students' learning experience.
Key elements of the scheme include:
This scheme demonstrates the end point of evaluation, and indicates where much of the evaluation of courses and subjects at UTS could be focused.
A fully integrated, value adding, teaching development and quality assurance scheme for the Course/subject level is outlined in Figure 1. The central focus is the course or unit of study and the central concept is change (for improvement) through a process of reviewing a course dossier and acting on the results of the review.
The responsibility for, and ownership of, development of the learning environment rests with the staff (individual or group) responsible for the course. Their task is to maintain a course dossier, and to use the contents of that dossier to maintain and/or improve the learning context for students. The review process would normally occur annually, and be based on a rolling data collection cycle. The period of the cycle would be determined by the group to fit the needs of their course and the timing of external requirements (such as 7-year course re-accreditation cycles).
The information in the dossier must be sufficient to answer a set of basic questions about the course. Key components in the process, including the criteria to be addressed are listed in 2.1-2.3 below. Sources of the information will be left to the discretion of the group with course responsibility. Some examples of sources and methods and their relation to the criteria are listed in Section 4. The university will assist in the data collection process by making available all the relevant data collected at that level.
2.1 Criteria for data collection
Sufficient data is to be collected to address the following three questions.
(i) What are the students' views of the course and/or its subjects on:
(a) assessment
(b) workload
(c) teaching
(d) clarity of goals and standards
(e) choice/freedom/independence
(f) overall satisfaction?
Figure 1: A Teaching Development and Quality Assurance Scheme
(ii) What are the employers' or other appropriate external views of the course on:
(a) the development of transferable/lifelong learning skills (such as flexibility, creativity, ability to communicate, work in a team)
(b)its relevance to industry/professions/life?
(iii) What are the views of the School/Department Board on:
(a) the relevance of each subject to the course
(b) the appropriateness of assessment
(c) the currency/appropriateness of content
(d) where and how the course objectives are met?
2.2 Possible contents of the dossier - sources of data
The following information should be contained in the course dossier
Assessment results
Graduate careers survey results (lifelong learning and CEQ)
Student entry data
Student outcome data
Student satisfaction surveys
Report from Vice-Chancellor's forum (where appropriate)
Course accreditation information or re-accreditation review
2.3 The discussion and review phase
Using the data in the dossier, the group with responsibility reviews the course annually, and from a supplied list of prompting questions, compiles a short report of the process including suggestions for changes, and the reasons why changes were recommended or rejected. This report and reports of action subsequently taken would be included in the dossier.
Monitoring of this process is to be kept at the appropriate Faculty-School or cross Faculty-School level. Every 3 years a group of staff established by the appropriate administrative body will audit the process for each course or collection of courses. This process would in turn be reviewed during the Faculty Developmental Review. Both processes would focus on the developmental aspects and the ways the information collected is acted on (the prompting questions), not necessarily the results obtained from the individual collection methods.
The main task of the group with course responsibility is the review and discussion of the information in the dossier, with the aim of bringing about changes if they are required. Such a scheme involves more work on the discussions, reviews and reporting than the current suggestions, but the focus on development, and information supplied by the university means a reduction in the overall work required to collect data at the faculty/school level. This change is expected to not only involve less work for the Faculties, but the work which is still required is more likely to add value to the education system.
4. Some Methods of Data Collection
At least one method from each of the three groups is required to address the three questions.
Students
Course Experience Questionnaire ((i)a-e)
Student Feedback on Subject Questionnaire ((i)a-f)
Discussions/meetings with students ((i)a-e)
Overall rating question to students ((i)f)
Suggestion boxes ((i)a-f)
Student diaries/journals ((i)a-f)
Interviews ((i)a-f)
External
School Advisory Committee ((ii)a-b)
Information from professional bodies ((ii)a-b)
Information from Disciplinary reviews ((ii)a-b)
Review of examples of student work ((ii)b)
Interviews ((ii)a-b)
Discussions/meetings with external people ((ii)a-b)
Academic colleagues
School Board meeting ((iii)a-d)
Peer evaluation questionnaires ((iii)a-d)
Review of examples of student work ((iii)b-d)
Interviews ((iii)a-d)