We believe there are at least three reasons:
Everything we do which impacts upon the students will influence their learning. Finding out how they experience their learning environment is the first step in improving that environment. Course and subject design, assessment strategies, availability of resources, integration of courses and subjects within a broader program of study, and many others are components of the learning environment and as such are valid matters for evaluating, refining and improving.
Tenure committees, promotion panels and selection committees require non-anecdotal evidence of teaching performance. This can be obtained using the methods described in this booklet. Note that the quality of teaching is also demonstrated by how we respond to the results of evaluation. Feedback statistics, comments from students or colleagues may serve as indicators of the value and validity of what we are doing but it is whether we act upon these indicators and how we act upon them that gives us a real opportunity to demonstrate the quality of our teaching.
In 1997 it was agreed that all teaching staff would undertake a regular evaluation of their teaching. The purpose of the evaluation is developmental and aimed at improvements in student learning. The methods chosen should be appropriate to the individual staff member's teaching context and particular focus. Evaluations may involve standard instruments provided by the Quality Development Unit, but should not concentrate solely on face-to-face teaching.
Quality Development Unit offers staff a questionnaire-based feedback system which includes the production of questionnaires, optical reading of student responses and the production of a summary report of the quantitative data. In view of the now widespread use of the Course Experience Questionnaire (CEQ) as an indicator of course quality, we have decided to offer staff at UTS a version of this questionnaire which we call the Subject Experience Questionnaire (SEQ) which is suitable for use in collecting feedback on either teaching or subject. The advantages of using the SEQ are that results are likely to give staff and academic units an indication of current students' perceptions of the teaching and learning areas which are surveyed annually using the CEQ with graduates.
The SEQ contains 25 items making up six scales: good teaching,
clear goals and standards, appropriate workload, appropriate assessment,
generic skills and overall satisfaction. We are proposing, that
with the use of either one of two overall ratings questions, one
on teaching the other on subjects, the questionnaire is appropriate
for use as either a teaching questionnaire or subject feedback
questionnaire. If you would like further information please contact
QDU staff.

An evaluation usually involves an analysis of data collected from more than one source and/or by more than one method. In the following pages, we outline a range of data collection methods for use in the evaluation of teaching and courses for which support can be obtained from QDU. There are also four common sources of data available.
Students are the obvious and most common source of data in evaluations of teaching and courses. However they cannot comment validly on all aspects of the process such as the currency of course content, so additional sources are usually required.
Colleagues include teaching partners, industrial partners, workplace supervisors, members of staff of academic development units, and they could also include staff from other faculties and universities.
Graduates and employers are important external sources. As these are often widely dispersed groups questionnaires are the simplest way of seeking their opinions. It is important to choose questions well as some employers will have never experienced university courses. Also graduates who have been in the workforce for several years may be well placed to comment on the objectives and outcomes, but are less well placed to comment on the processes used to achieve these objectives. Current students and recent graduates are probably better placed to comment on the way courses are run/administered.
Large amounts of documented information on teaching and courses
are now produced regularly for a variety of purposes. This material,
which includes: subject guides, distance learning materials, educational
software, Web-based materials, Course Committee and other board
minutes, internal working papers, external examiners' reports,
Department research reports and staff development programs etc.,
disciplinary reviews, reports from professional bodies, graduate
destination surveys, and student satisfaction surveys; are usually
freely available for analysis. Students' work, for example assessment
tasks, project reports, etc. provide evidence of learning.
Questionnaire items can be closed (containing tick box items) or open (asking students to write their responses). Questionnaires using open ended questions (Gibbs et al, 1988) usually elicit more detailed responses, but are sometimes more difficult to analyse. Entire questionnaires using open ended questions can be specifically designed to collect data about teaching, subjects or courses, or alternatively questionnaires can contain some open and some closed items.
The QDU offers all combinations of open and closed questionnaires. To obtain student feedback on your subject, course or teaching by questionnaire:
QDU will generate individualised questionnaires for each class and send you the number of copies you request for your students.
Included with the questionnaires we send you is a guide to how the questionnaires might be distributed and collected.
The student feedback results will be returned to you in two forms:
Both forms of results will be returned after the assessment period for your subject.
It is important to remember that students can comment validly only on some aspects of teaching. Students' feedback is valuable input into any review, but your own reflection on what you want students to learn and how you are helping them to learn, and your peers' candid comments are even more important.
Note that there is the facility to add up to five open ended questions (asking for a written response) on all of the standard questionnaires A-H described below.
Subject Experience Questionnaire (SEQ)
The SEQ is an adaptation of the CEQ which is used nationally as a performance indicator. Both contain items making up five scales: good teaching (Items 3, 7, 15, 17, 18, 20) clear goals and standards (Items 1, 6, 13, 24) appropriate workload (Items 4, 14, 21, 23) appropriate assessment (Items 8, 12, 16, 19) generic skills (Items 2, 5, 9, 10, 11, 22). We are proposing, that with the use of either one of the two overall ratings questions listed as Item 26, the following 25 items are appropriate for use as either a teaching questionnaire or subject questionnaire.
Closed Response items
Open ended items
This questionnaire contains the 14 agree/disagree items and 1 overall rating question. There is scope to add up to 10 agree/disagree items and 3 overall rating questions of your choice.
(see your school/faculty administrator)
We can prepare a questionnaire with the same questions as you have requested in the past, if you let us know the following details about that questionnaire:
Prepare your own questionnaire from the supplied Itembank.
QDU can offer support for staff interested in developing questionnaires to send to groups such as employers/graduates or academic peers. For further information, contact QDU.
Group discussions have the potential to be just as informative and are considerably easier and quicker to prepare and run than the administration of questionnaires.
Informal group discussion such as asking a group "How's the course/subject/lecture going?" can be valuable in helping make running alterations to subjects but is limited in the depth of response it can elicit and should be kept brief and not overused.
Structured discussion groups can provide more detailed feedback on a subject. The aim of the discussion group is to collect issues from the small group (usually 5-15) and to discuss a few target issues in depth. You can facilitate these discussions yourselves so as to hear first hand what the students have to say, or have them run by a trusted colleague. Examples of these activities include Focus Groups and Questionnaire-based Discussions. A highly structured form of focus group is the Nominal Group Technique (Gibbs, 1982) which is explained further below.
Example: the Nominal Group Technique
Nominal Group Techniques are so called because at various points in the process the members of the group are acting as individuals and therefore the group is a group in name only.
Nominal Groups usually work in a number of steps:
The proforma below can be modified and distributed to students
for use in this process, or it can be adapted for use in less
structured focus groups.

As we noted in the section on data sources, there are a variety of materials now available for analysis. The form of that analysis will depend on the aim of the evaluation. An example based on descriptions of the assessment system is given below. The method can be used with any of the materials in the data sources list described earlier.
Example: assessment audit
Assessment is the main activity which guides students through a subject, it signals to them what is important, directs them as to how they should approach various aspects of the subject and determines the approach they take to learning within the subject. An assessment audit is a quick convenient way to review assessment procedures. The proforma below provides a list of things to consider and questions which may be asked.
Example proforma for assessment audit
Assessment methods
Examine the assessment methods used for this subject. What do
they assess?
List what you think students would need to do to pass each assessment task. What would they need to do to excel? Are these appropriate activities? Do they encourage appropriate approaches from the students?
Objectives
Does the assessment test the educational objectives of the subject?
Review how the assessment helps students achieve these objectives.
How and what does it help them learn?
Focus
Review the assignments set for each element of the subject. Is
the workload for any element significantly heavier or lighter?
Do the marks reflect this?
List the objectives tested by the components of the final exam/major essay. Do any elements appear more frequently than others? Is this appropriate?
Weighting
Are the assessment tasks of an appropriate level of difficulty
for students at that point in the course and subject?
Outcomes
Scan the distribution of assessment results for the subject. Is
it appropriate? If not why not?
Look at the essay, project or final exams of students who are on the borderline of pass/fail or distinction/high distinction. Does the work match the mark? What are the main differences between the work of each group? How can poorer students be helped? Has the assessment approach been criterion referenced?
Feedback
How effective are the processes for supplying feedback to students?
Having a colleague observe and comment on your teaching can be a very useful way of obtaining feedback. However care should be taken to make this a productive experience for all concerned. Teaching is often a matter of style and it is possible to get distracted by stylistic differences rather than focussing on the real issues of student learning. In order to help your colleagues focus on those aspects of teaching and learning which are most important to you it is best to provide them with an observation sheet or checklist such as the one below. The checklist should focus on events rather than opinions, it should contain a record of what the observer saw happening not whether they thought this was good or bad. At the end of the session the checklist should be discussed with the observer to clarify points and more fully explore what was going on in the session
Note that colleagues are not only those people working in your department but people from academic development units, people from other schools, faculties or institutions, even experts from industry.
Example: colleague observation checklist
Responsibility
Who made the decisions about the content of the session? Who made
the decisions about the processes involved? Who controlled the
session?
Teaching strategies
What teaching strategies were employed? (Uninterrupted telling,
dialogue with students, overviews and summaries, questioning students,
answering questions, setting problems) What appeared to be the
intentions behind these strategies?
Learning strategies
What strategies did the students employ to aid their learning?
(Note taking, questioning, passive listening, talking to other
students, reviewing previous notes or the text book) Who initiated
these strategies? What type of learning might result?
Feelings
What expressions or evidence of feelings of students or the teacher
were evident? (Boredom, anxiety, distraction, anger ...)
Problems
What problems were encountered during the session? What critical
incidents occurred, how were they handled?
Interest/attention
Where were students directing their interest/attention.
Interviewing students individually can be very revealing and helpful and is highly recommended as a way of finding information that no other method can reveal. It is a time consuming process and can be intimidating for the students involved. It should never be attempted without their full cooperation.
If you wish to use this method, IML has information on questions to ask.
Your own thoughts are a key evaluation component. It is rare for us not to finish a teaching session without some feeling about how the session went - whether it went well or not so well, what worked what didn't etc. It is equally rare for us to record these thoughts and feelings. Yet they are a valuable source of information on our teaching and if we don't record them somewhere we quickly forget them. A written record of experiences and reflections upon them can be kept using a reflective journal or teaching portfolio (Appendix 1). Use of the proforma below is also a simple way of preserving these thoughts. These forms, filled in regularly and kept together, form a useful resource in any future planning of teaching sessions. Reflective analysis of teaching and subjects can also occur verbally through debriefing sessions with colleagues.
Example: proforma for self evaluation

The products of assessment are a readily available source of quantitative and qualitative learning outcomes data. Pass rates, average marks and the spread of marks around that average, can all tell you things about how students are dealing with their subjects (Rowntree 1987). An analysis of the quality of students' work, especially the work that is near to pass/fail or near to distinction/credit boundaries tells us a lot about how successful teaching has been. One method of analysing the data qualitatively is to use the SOLO Taxonomy described below.
Example: use of SOLO taxonomy to analyse assessments
In 1982, Biggs and Collis described five structural levels of learning outcomes which ranged from incompetence to expertise (Figure 1). This is now known as the SOLO (Structure of the Observed Learning Outcome) taxonomy. The first level (incompetence) was labelled Pre-structural and is applicable to an outcome containing nothing of relevance to the knowledge in question. The second level is Uni-structural, and includes outcomes where there is a reference to only one relevant item. Multi-structural outcomes are those where more than one relevant item is included, but those items are listed or described independently. Outcomes of a Relational (level four) nature do not necessarily include a greater amount of knowledge than in the case of Multi-structural outcomes. In Relational outcomes the understanding is integrated and related, and the separate elements are described as part of an overall structure. The final, and most complete level (Extended Abstract) includes those outcomes that demonstrate the generalisability of the understanding to new contexts. Students with this understanding are able to draw upon it in (some) new contexts.
Figure 1: Five structural levels of learning outcomes

Students written responses, from paragraphs to whole essays, can be analysed using the SOLO framework. For example a response that involved a list of things a student knew in response to a question (a multi-structural response) is usually of a lower quality than a response where the student displays a knowledge of the relations between these things (relational response). Where a teacher was aiming for learning outcomes of a relational nature, large numbers of multistructural responses would be sufficient reason to question the perceptions students had of their task.
For more details/examples of this method, contact IML.
We believe the most important use of feedback data is in the improvement of teaching and learning. In the information we send you with your questionnaire summary reports, we include advice on interpreting the results including ways to improve teaching and learning.
As illustrated in the overview diagram on page 3 there are many other uses to which the results of evaluation can be put. There is not sufficient space to address all these issues here, but it is vital that evaluation data are put into context no matter in what way they are used. In this booklet we describe two ways of doing that. The first (for your personal use) is in an academic portfolio (Appendix 1). The second (for departmental or faculty use) is in a course portfolio (Appendix 2). Please contact IML staff if you wish to have more information on other ways of using evaluation data.
A Portfolio is an organised record of academic experiences, achievements and professional development over a period of time. It consists of a collection of documents which illustrate the variety and quality of work done, along with reflections or reviews on these documents. A portfolio can be kept in a variety of ways: in a folder with sub-sections, in a series of files in a filing cabinet, or in electronic form with a file of supporting documentation.
The guidelines in Appendix 1 suggest ways that you can develop your Academic Portfolio in the area of teaching and educational development. As each academic's experience and context will be different there is no expectation that all Academic Portfolios will be similar. Rather they should contain evidence of individual academic achievements and experiences, with reflections on these experiences. Appendix 2 contains information on the use of a course portfolio. The one thing that all portfolios have in common, and the one thing that differentiates them from what would otherwise be no more than a collection of materials and reflections, is that the content of the portfolio is linked to and integrated with an articulation of a teaching/learning role or philosophy, or with course aims and objectives or graduate attributes.
Armstrong, J. and Conrad, L. (1994) Subject Evaluation: A Resource Book for Improving Learning and Teaching. Griffith Institute for Higher Education, Griffith University, Queensland.
Biggs, J.B. (1991) Teaching for Learning: The view from cognitive psychology. Hawthorne: Victoria, Australian Council for Educational Research.
Gibbs, G (1982) Eliciting Student Feedback from Structured Group Sessions. Oxford: Oxford Polytechnic, Educational Methods Unit.
Gibbs, G., Habeshaw, S., and Habeshaw, T. (1988) 53 Interesting Ways to Appraise Your Teaching. Bristol: Technical and Education Services.
Gibbs, G. (1989) Creating a Teaching Profile. Bristol: Technical and Educational Services.
Ramsden, P. and Dodds, A. (1989) Improving Teaching and Courses: A Guide to Evaluation. Parkville: Centre for the Study of Higher Education, The University of Melbourne.
Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge (Chapter 11).
Rowntree, D (1981) Developing Courses for Students. Maidenhead: McGraw - Hill.
Rowntree, D. (1987)Assessing Students: How shall we know them?. London: Kogan Page.
What should an Academic Portfolio include?
Example: Alison's portfolio
|
This is what her portfolio contains by the end of her first semester: Overview and personal information
Teaching
Scholarship and research
Service to the university and the community
|
Getting started
Developing a portfolio is an ongoing process. As your portfolio develops over time, you will include documents which provide evidence of a wide range of activities and achievements, along with your reflections on these documents. Getting started means beginning to create a framework and sections for your portfolio which you find useful and can continue to use, then adding one or two items at a time as you engage with different aspects of your work. "Alison's portfolio" (see box) shows one example of the kinds of sections and documents that could be compiled over the course of a semester.
There is no single right way of keeping a portfolio. Whatever method you use, you should try to add to your documents and reflections on a regular basis, both to assist your professional development and so that your portfolio will be up to date for occasions when you discuss your academic progress or are planning an application for promotion.
In the rest of this section we provide specific advice on developing only the teaching and educational development section of a portfolio. We have included some brief advice in three categories:
The ideas described are suggestions. Your portfolio will be individual and is likely to have more emphasis in some areas than in others. Remember that an effective portfolio will include both documentary evidence and reflection, eg a subject outline that you have developed with a reflection on the effectiveness of the outline in terms of student learning and the changes you may make as a result.
What is reflection?
Reflection is a process of reviewing and thinking critically about our activities and the world around us. It goes beyond describing what we do, to thinking about why we do things, whether they have gone as intended, why we think they worked well or didn't and how we might do them differently next time. Recording these reflective thoughts can help us to crystallise our understanding of experiences, make connections between different aspects of our work and identify possible improvements. Recording personal reflections has the added benefit of documenting learning over time through recognition of the changing nature of our reflections.
Reflection includes constantly testing the assumptions and actions related to our work. Brookfield (1995) describes reflective teaching in the following way:
"Critically reflective teaching happens when we identify and scrutinise the assumptions that undergird how we work. The most effective way to become aware of these assumptions is to view our practice from different perspectives. Seeing how we think and work through different lenses is the core process of reflective practice." (Brookfield, S. (1995) Becoming a Critically Reflective Teacher. San Francisco: Jossey Bass, p.xii-xiii)
In the case of teaching, the different "lenses" that Brookfield refers to include personal beliefs and the perspectives gained from students, colleagues, and reading the literature. Reflecting on the evidence that you collect from a range of different sources is an important component in creating your Academic Portfolio.
The private and public faces of the portfolio
It is quite reasonable to keep some aspects of the academic portfolio private. There are likely to be some reflections or documents which are important for your development but which you do not wish to show to others.
A well developed academic portfolio would be too much to present to, for example, a promotion committee. It would be more appropriate to include an extract with descriptions of and reflections on your major achievements and references to other documentation.
Teaching and educational development
Many academics spend a great deal of time preparing for teaching, teaching and assessing their students' learning. We hope that developing this section of the portfolio will enable you to improve your practice as a teacher through reflection and self evaluation.
Imagine that you are trying to give someone else a picture of your teaching - what you do and why you do it that way. Your portfolio should create a picture that illustrates some of the complexity and variety of what you do and why. At minimum, your portfolio would include a statement outlining your own teaching philosophy, and an overview of your teaching experience: the range of subjects and classes you have taught, postgraduate supervision etc. It would then include some items documenting different aspects of your teaching.
Your teaching philosophy- what teaching means to you
Firstly it would be a good idea to describe your own understanding of teaching and the way in which you see the relationship between teaching and learning. In this way you will have a record of your thoughts on teaching and learning as you commence your career as an academic at UTS and you will be able to return to your initial ideas from time to time and consider the development and changes that have occurred as you have developed your expertise. This section could be private to begin with, but later you might develop a description of your teaching philosophy to include in the portfolio extract that you would show to others.
Supporting evidence
Examples that could be used to support or provide evidence about your teaching over time may include:
Questions for self evaluation
Introduction
In late 1994 UTS adopted a teaching development and quality assurance scheme based around a course portfolio to assist in the improvement of the students' learning experience. Comments from staff who have been using this scheme are recorded on http://www.clt.uts.edu.au/courseportfolio.html
This scheme demonstrates the end point of evaluation, and indicates where much of the evaluation of courses and subjects at UTS could be focused.
A teaching development and quality assurance scheme
A fully integrated, value adding, teaching development and quality assurance scheme for the course/subject level is outlined in Figure 2. The central focus is the course or unit of study and the central concept is change (for improvement) through a process of reviewing a course portfolio and acting on the results of the review.
The responsibility for, and ownership of, development of the learning environment rests with the staff (individual or group) responsible for the course. Their task is to maintain a course portfolio, and to use the contents of that portfolio to maintain and/or improve the learning context for students. The review process would normally occur annually, and be based on a rolling data collection cycle. The period of the cycle would be determined by the group to fit the needs of their course and the timing of external requirements (such as 7-year course re-accreditation cycles).
The information in the portfolio must be sufficient to answer a set of basic questions about the course. Key components in the process, including the criteria to be addressed are listed below. Sources of the information will be left to the discretion of the group with course responsibility. Some examples of sources and methods are described earlier in this book
Criteria for data collection
Sufficient data is to be collected to address the following three questions.
(i) What are the students' views of the course and/or its subjects on :
a) assessment
b) workload
c) teaching
d) clarity of goals and standards
e) attainment of graduate attributes
f) overall satisfaction?
(ii) What are the employers' or other appropriate external views of the course on:
a) the development of graduate attributes (such as flexibility,
creativity, ability to communicate, work in a team)
b) its relevance to industry/professions/life?

Figure 2: A Teaching Development and Quality Assurance Scheme
(iii) What are the view of the school/department board on:
a) the relevance of each subject to the course
b) the appropriateness of assessment
c) the currency/appropriateness of content
d) where and how the course objectives are met?
Possible contents of the portfolio - sources of data
The following information could be contained in the course portfolio
The discussion and review phase
Using the data in the portfolio, the group with responsibility reviews the course annually, and from a supplied list of prompting questions, compiles a short report of the process including suggestions for changes, and the reasons why changes were recommended or rejected. This report and reports of action subsequently taken would be included in the portfolio.
The main task of the group with course responsibility is the review and discussion of the information in the portfolio, with the aim of bringing about changes if they are required.
Item types
There are three types of item in the itembank:












QDU offers all combinations of open and closed questionnaires. To obtain student feedback on your subject, course or teaching by questionnaire:
QDU will generate individualised questionnaires for each class and send you the number of copies you request for your students.
Included with the questionnaires is a guide to how the questionnaires might be distributed and collected.
Both forms of results will be returned after the assessment period for your subject.
It is important to remember that students can comment validly
only on some aspects of teaching. Students' feedback is valuable
input into any review, but your own reflection on what you want
students to learn and how you are helping them to learn, and your
peers' candid comments are even more important.