Information for UTS staff on
Assessment
Compiled by Keith Trigwell for the Working Party on Assessment, Autumn Semester, January 1992.
Information for UTS staff
on
Assessment
Compiled by Keith Trigwell for the
Working Party on Assessment, Autumn Semester, January 1992.
Composition of the Working Party: Professor
Brian Low (Chair), Pro-Vice-Chancellor (Academic Support); Professor
Christine E Deer, School of Teacher Education; Rolf Kater, Student
Association; Associate Professor Peter Logan, School of Physical
Sciences; Dr Keith Trigwell, Centre for Learning and Teaching.
Contents
A Note for
UTS Staff
ASSESSMENT ISSUES
Why Assess
Students?
Assessment and Learning
Over-assessment
Statistical Issues in
Assessment
Other Issues
FORMS OF ASSESSMENT
Multiple-choice
Questions
Essays
Short Answer Questions
Alternative Examinations
Learning Contracts
Peer Assessment
Oral (Viva voce) Examinations
Seminar Presentations
Other Forms of Assessment
ASSESSING PARTICULAR LEARNING EXPERIENCES
Assessing Work
Experience
Assessing Laboratory
Work
Projects and Group Work
Assessing Problem-based Learning
Assessing Mastery Learning
COMPLETE ASSESSMENT PACKAGES
KEY REFERENCES
A
Note for UTS Staff
This publication has been prepared for UTS staff by the Working Party on Assessment. It is presented initially as part of the Report from the Working Party to Academic Board, but is intended for use as a stand-alone resource book for UTS staff.
The contents draw heavily on previous writings on assessment, on the assessment practices of some sections of UTS, on contributions from staff of UTS, and on submissions to the Working Party. These contributions are acknowledged within the text.
This document should also be considered in conjunction with two publications which overlap with and extend the issues addressed here. Both of these publications form a part of this information package on assessment and a copy of each is being supplied to each School. They are:
Andresen, L., Nightingale, P., Boud, D. and Magin, D. (1989). Strategies for Assessing Students (Teaching with Reduced Resources, No. 1.) Professional Development Centre, University of NSW.
Gibbs, G., Habeshaw, S. and Habeshaw, T. (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol.
The publication is arranged in six parts. The first part addresses three issues: the relations between assessment and learning, over-assessment and ways to overcome it, and statistical issues in assessment.
The second part contains a description, advantages, disadvantages, procedures and examples of the common forms of assessment used at UTS, or forms which, if not commonly used at UTS, may be of interest to staff. The third part looks at assessing particular learning experiences.
The fourth part addresses the relations between objectives for a subject and the whole assessment package for that subject, and includes UTS examples.
Parts five and six contain a glossary of assessment terms, and a summary of the references.
Further information on assessment is available
from the Centre for Learning and Teaching.
Why
Assess Students?
There are two main reasons for assessing students. The first is so that staff can determine the level at which a student is performing in terms of knowledge and understanding, skills and, in some subjects and courses, attitudes. The appropriate skills, knowledge and understanding are described in the objectives set for the subject, these objectives being provided to students at the beginning of study of the subject. An appropriate set of assessment tasks is one which, in the professional judgement of the lecturer, allows students to demonstrate their level of performance in achieving the objectives of the subject. This type of assessment, when used to determine grades or marks is known as summative assessment.
The second reason is to give students feedback on their progress towards the development of knowledge, understanding, skills and attitudes. This type of assessment is known as formative assessment.
Assessing students' knowledge, understanding, skills and attitudes is an extremely difficult task. In general, the instruments we use to assess students tend to be limited in what they are able to reveal. We are further constrained by limitations such as having to assess whole groups at certain times in ways that lead to qualitatively different grades, and all the methods used are compromised because of limited financial and human resources. We must accept that none of the methods we currently employ are totally satisfactory.
What this amounts to is that we need to retain an open mind in our interpretation of assessment information and vigilance in our use of these methods so that we do not read more into the information we collect than we can reasonably expect using those methods.
This publication aims to address some of
the issues affecting the quality of our assessment strategies.
It also contains summaries of the major types of assessment, examples
of their use in UTS and sources for further information.
Assessment
and Learning
The following extract contains a brief overview of the literature on the relations between assessment and learning. This section is included because the learning side of assessment is often neglected when resource pressures force assessment methods to focus more on accreditation than on learning.
There are two main purposes of student assessment. The first intends to improve the quality of learning. Students engage in the problems and discourse of a given area and are given encouragement, response and feedback on what they do, as appropriate, with a view to them becoming more effective in their learning. This is formative assessment, or assessment for learning. The second concerns the accreditation of knowledge or performance: students are assessed to certify their achievements. This occurs primarily for the award of a degree or diploma, though various components of assessment are usually taken into account in making this judgement. This is summative assessment, or assessment for the record.
In both cases judgement is involved, but in the first it directly serves the needs of the student and in the second it primarily serves the needs of the external world. Assessment also contributes to motivation through the recognition of achievement. However, the relationship between certification and motivation is a complex one. For as many high-achieving students who are encouraged and stimulated by their high grades, there are others who are discouraged and alienated by their lesser grades. Grading per se is not a motivator and can only be used as such with very great care in any given situation. Assessment is also used for various administrative reasons, such as for the allocation of students to particular groups, but these should be regarded as secondary purposes.
Students tend not to learn well if we are not effective in the former, and they cannot be recognised as competent if we neglect the latter. Unfortunately, resource pressures increasingly lead us to protect assessment for accreditation at the expense of assessment for learning. Learning is so driven by assessment that the form and nature of assessment often swamps the effect of any other aspect of the curriculum.
While there may not be a current public debate about assessment, over the past 20 years an interesting literature has been emerging in higher education on the relationship between assessment and learning. It is not the place here to review it in great detail, but research has shown the following features.
(1) Students are assessed on those matters on which it is easy to assess them, and this leads to an over-emphasis on memory and lower-level skills (e.g. Black, 1969). Creating questions which test higher order skills is not impossible, but it demands a degree of professional commitment to test design which is absent from many departments.
(2) Assessment encourages students to focus on those topics which are assessed at the expense of those which are not (e.g. Elton & Laurillard, 1979). In other words, assessment tasks define the syllabus, and, if students want to get good marks, they focus on these aspects at the expense of others which might capture their interest.
(3) The nature of assessment tasks influences the approaches to learning which students adopt (e.g. Ramsden, 1988). Not only does the content of assessment define what is to be studied, but also the kind of task required shapes the learning strategy of students. If students perceive reproduction of information to be rewarded, they will emphasise memory work, and if they see problem-solving emphasised, they will tend to practise solving problems.
(4) Students who perform well in university examinations can retain fundamental misconceptions about key concepts in the subjects they have passed (e.g. Dahlgren, 1984). Some of the most profoundly depressing research on learning in higher education has demonstrated that successful performance in examinations does not even indicate that students have a good grasp of the very concepts which staff members believed the examinations to be testing.
(5) Students give precedence to assessment which is graded (e.g. Becker et al., 1968). Grading acts as a kind of currency indicating what teachers value. It is in the best interests of students to focus on those things which produce the greatest return.
(6) Successful students seek cues from teachers to enable them to identify what is important for formal assessment purposes (e.g. Miller & Partlett, 1974). Effective performers often use the strategy of attending lectures in order to obtain cues about what of the vast range of matters in a given subject will be emphasised in examinations. They focus their energies on these and may spend significantly less time in studying than their less successful peers.
The picture painted by this research is bleak. Despite the good intentions of staff, assessment tasks are set which encourage a narrow, instrumental approach to learning that emphasises the reproduction of what is presented, at the expense of critical thinking, deep understanding and independent activity.
These findings indicate effects which are quite contrary to those which are sought. Students are discouraged from taking initiatives beyond their lecturer's interpretation of syllabus, and they spend their time 'swotting for examinations' rather than trying to internalise and make sense of the subject. Evidence such as this suggests that very great care must be exercised in the selection and implementation of assessment asks, otherwise they can have counter-productive results.
Becker, H., Geer, B and Hughes, E.C. (1968). Making the Grade: The academic side of college life. New York: Wiley.
Black, P.J. (1969). University examinations, Physics Education, 3, 93-99.
Dahlgren, L.-O. (1984). Outcomes of Learning. In F. Marton, D. Hounsell and N. Entwistle (Eds) The Experience of Learning. Edinburgh: Scottish Academic Press.
Elton, L. and Laurillard, D.M. (1979). Trends in research in student learning, Studies in Higher Education, 4, 87-102.
Miller, C.M.L. and Partlett, M. (1974). Up to the Mark: A study of the examination game. Guildford, SRHE and Open University Press.
Ramsden, P. (1988). Studying Learning, Improving Teaching. In P. Ramsden (Ed) Improving Learning: New Perspectives. London: Kogan Page.
Boud, D. (1990). Assessment and the Promotion
of Academic Values, Studies in Higher Education, 15, 101-111.
Over-assessment
In many sectors of Higher Education there is a general tendency to over-assess students. Some of these practices arise from a desire to increase reliability in the test and to encourage students to study widely for the course. However, over-assessment increases staff time needed to run the system and does not necessarily lead to greater student learning. This section contains some information on avoiding over-assessment. It is taken from one section of the booklet Strategies for Assessing Students which is being circulated to all Schools. It is presented here also as an example of what the booklet contains. Other strategies related to reducing the demands of assessment when there are fewer resources are listed at the end of this section.
First consider
Then consider in the particular assessment
task being implemented,
After these considerations, the items that follow may provide a source of ideas for you to use in starting to develop a more economical assessment system.
Assessment demands have a major impact on both student work-load and on staff resources. It is often possible to reduce the number of tasks for assessment without detracting from the rigour of a course or diminishing the reliability of final assessment grades.
In a study of assessment loads at Macquarie University large variations were found from course to course, and some subjects were able to be pin-pointed as clearly being over-assessed by comparison with others.
We should try to estimate the number of hours spent by staff in setting, explaining, supervising, collecting, marking, grading, processing scores and giving feedback, all relative to the number of contact house of the course.
A similar exercise can be carried out in regard to student loads. There we can compare the number of hours typically spent being briefed upon, carrying out, and receiving feedback on assessment tasks, relative to the total contact hours of the course.
Subjects with anomalous assessment loads on either students or staff become, by definition, cases for suitable attention.
Macquarie University Education Committee (1984) 4-5, 19-21
It is wise to try to anticipate where students will take the questions we set - essay questions in particular. Some instructions, some questions, may subsequently make our task as markers much more difficult than we ever intended. It is obviously best to discover this fact before students go to work on it.
This involves taking time to put ourselves in a student's position and imagine some of the possible mis-interpretations of our question - and what those might lead to when we finally get back the piece of work for marking. This can be difficult when it concerns questions we have ourselves set, and other staff can often given the best insight into ambiguities and lack of precision in our instructions and question-wording.
Nightingale (1986) 18-27
Provided group work is compatible with the aims of assessment in a particular case, it can often be most efficient and economic if group assessment tasks are both done and marked in class time.
A seminar group, for instance, can handle a question in class by collaboratively drafting not-form or outline answers. The result can then be appraised in class.
Either approach can lead to better discussion and clarification of writing or problem-solving strategies and marking standards.
Gibbs et al. (1986) 103-107; Nimmo (1977) 186-187
These refer to any kinds of tasks that may lead up to the major one. Where prior tasks on this kind are set, it is often reasonable to grade only the last, and the earlier ones may not need to be assessed at all in the conventional way.
For examples of these prior-stage tasks, we should consider - among other more obvious types - the use of course journals, reading logs, and reflective pieces of many different kinds.
Nightingale (1986) 10-13; Zubrick (1985); Gibbs et al. (1986) 115-117
Assessment that is "formative" in function - i.e. not required in order to give a final grade- can often be carried out on lead-up tasks, such as synopses of proposed essays.
In such instances it should be remembered that the appraisal of these does not necessarily have to be done by academic staff. Students can given each other constructive feedback and learn a lot as they do so, provided the task is designed and set with this option in mind.
Paxton (1976)
Short answers, lists, synopses and outlines can be used to replace extended essays in many cases. If we critically inspect what really needs to be assessed in our subjects, this might be only a portion of what would be demonstrated in a full, complete essay.
We need to ask why, then, we should have to mark a full essay if much of what it reveals is superfluous to our present assessment needs?
The main arguments against giving a choice among essay-type question options in tests and examinations have nothing to do with economics or efficiency in the first instance. They are that, according to research, if a paper has say 12-15 questions of which only 5 are to be chosen, the chances are that:
(i) the different options will vary in difficulty,
(ii) marker subjectivity and unreliability will be greatly amplified, and
(iii) many students will choose unwisely among options, not doing themselves justice.
Hence on all empirical counts the practice of giving a choice tends to make essay examinations unreliable and unfair. Each student in fact sits for a different paper.
On the issue of resource efficiency, common sense indicates that we have less work to do if ...
(i) fewer questions need writing when the paper is being set, and
(ii) fewer marking schemes need designing during the marking stage.
In addition markers generally find it more efficient to mark more answers of the same question than a few answers of many different questions.
So if you think you would find it easier to set papers comprising a set of common questions with no options, you have research and commonsense both on your side.
Students, accustomed to being given options, may resent the change as being unfair, so you may need to marshal your facts and explain clearly your reasons for the new policy. It will be a fairer exam for them, and the marks you award will be more reliable.
Stanton (1981)
Student surveys often come out strongly in favour of a diversity of examining methods because of the very reasonable belief that only in this way can a "fair deal" be assured.
Apart from any intrinsic value in using a diversity of assessment methods, a policy of encouraging diversity may also be consistent with gain in efficiency.
Whether this is so will depend, however, upon which particular formats are included in the package, since not all methods need be as time-consuming as others in setting and marking. For example, an economically diverse package may need to comprise only one long written task, supplemented by a variety of shorter, more easily marked tasks.
As an instance, the examination structure in one professional medical course involves ...
(i) a written paper using the modified essay question style,
(ii) a multiple-choice test,
(iii) a simulated diagnostic interview, and
(iv) a practice test.
Beard (1970) 193-196
Beard, R. (1970). Teaching and Learning in Higher Education. Harmondsworth: Penguin.
Gibbs, G., Habeshaw, S. and Habeshaw, T, (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol.
Macquarie University (Education Committee) (1984). Report of Working Party on Teaching and Assessment Methods.
Nightingale, P. (1986). Improving Students' Writing. Sydney: HERDSA Green Guide No 4.
Nimmo, D.B. (1977). The undergraduate essay: A case of neglect? Studies in Higher Education, 2, 183-189.
Paxton, S. (1976). Pre-submitting essays, re-attempting tests. University of Queensland, St Lucia: TEDI.
Stanton, H. (1981). Optional Question on Essay-Type Examination Papers. The University Teacher, 4 (2).
Zubrick, A. (1985). Learning through writing: the use of reading logs. HERDSA News, 7 (3), 11-12, 24.
Andresen, L., Nightingale, P., Boud, D. and Magin, D. (1989) Strategies for Assessing Students (Teaching with Reduced Resources, No. 1.) Professional Development Centre, University of NSW.
Other strategies in the booklet are
1. Deciding whose interests assessment is serving
2. Avoid over-sampling the course
3. Avoid over-questioning topics
4. Avoid over-reading student work
5. Avoid over-commenting on student work
6. Avoid over-grading student work
7. Refine current policies and use present methods better
8. Consider alternative approaches to assessment policy and practice
9. Consult original sources for how-to-do details
10. Get support, help or advice from an
Educational Development Consultant
Statistical
Issues in Assessment
Assessment in education is a statistical problem. Statistics can be defined as 'the art and science of making sense of numerical information'.
You do not have to be a statistician to carry out assessment in education, but it helps to remember that what you are doing is essentially something statistical. If you neglect the advice that statistical theory has to offer, you run the risk of making serious mistakes.
Statistical theory is not in some way opposed to a human, sympathetic and flexible approach to assessment. On the contrary, a careful statistical analysis of the problem shows the need for these very qualities.
To measure a student's performance, industry or potential in a certain subject area, either in absolute terms or relative to other students.
What are we trying to measure (for example, in intelligence tests)? Are we measuring in absolute or relative terms?
How can we tell if one test is a better measuring instrument than another? How do we get the best test, or the ideal method of assessment?
How do we make test marks from different situations or subjects comparable? How do we combine information from a number of different measurements?
What happens if we do not do anything statistical?
Criterion referenced assessment: how well has the student done by comparison with some predetermined criterion?
Norm referenced assessment: how well has the student done by comparison with the norms established by his or her colleagues in the whole group?
Another possibility is self referenced assessment: how well has the student done by comparison with their own earlier achievements?
Norm referenced assessment may be the most appropriate for public examinations such as the Higher School Certificate (HSC). With large numbers of students, the standard can be expected to be similar from year to year. One of the functions of the HSC is to rank students for tertiary entrance.
Criterion referenced assessment seems to be more appropriate for the typical university course. Numbers are smaller, so there is no guarantee that the standard is the same from year to year. We are comparing students with predetermined criteria - the course objectives - to determine whether they should be passed.
In practice there is not as much difference between criterion and norm referenced assessment as there seems to be. In criterion referencing, we are comparing a student against norms established by a larger number of other students at earlier times. With criterion referencing, we can sort students into multiple categories (for example, graded passes). With norm referencing, we can get information about how far away the student was from the pass/fail criterion.
We often use norm referenced techniques to obtain and combine marks from assessment events and then criterion referenced techniques to report the final results as Fail, Pass, and Graded Passes. We should be aware of the dual nature of this process and ensure that we keep the nexus between them by adjusting the marks so that 50% does in fact represent what we regard as a Pass (in subject where 50% is a pass) and similarly for the higher grades.
Validity is concerned with whether a test measures the ability we are trying to measure, or some different related or unrelated ability. (In statistical terms, is it an unbiased estimator of the student's ability?)
Validity is concerned with questions such as: Is my assessment of the student's behaviour correct? Have I really seen evidence of what I think I have? How confidently can I generalise from what I have seen? Can I predict anything of the student's educational future?
Reliability (or consistency) is concerned with whether the assessment measures with low variability between repeated situations. (In statistical terms, do repeated measurements have a high correlation?)
Reliability is concerned with questions such as: Would other assessors agree with my interpretation of the student's behaviour? Would I myself interpret his or her behaviour in the same way if I saw it again?
Ebel (p. 309): Expertly constructed educational achievement tests often yield reliabilities of 0.90 or higher. In contrast, the achievement tests used in many elementary, secondary and college classrooms often show reliability coefficients of 0.50 or lower.
Objective or multiple choice type tests are more reliable than subjective, divergent tests such as essay questions (typical reliability of 0.8 for re-reading by same rater, 0.7 for two raters, 0.7 for two essays from same candidate with one rater). Validity and reliability are opposed. The more structured the question, the less evidence we have on which to make valid assessments of the qualities we are interested in, but the more agreement we will get about our assessment. This is not to say that objective questions are totally objective. There is a subjective element in the choice of which questions to include.
Rowntree (p. 197): The more a student's performance approximates a 'work of the imagination', the less 'reliable' is assessment - unless assessors agree to ignore the contentious aspects and concentrate on the measurable. And the more we look for high quality in students, the more divergent and open to the exercise of the imagination are our assessment methods likely to be.
When given to a group of people, any test/examination/assessment task results in scores. These scores can be summarised by an average value (the mean) and a spread or measure of variability (the standard deviation).
Assessment tasks do not measure without error. With different questions, at different times and under different conditions, a student's score will be different. The standard error of measurement gives a reasonable plus or minus on each individual score. The value of the standard error of measurement depends on the standard deviation of the scores and on the reliability of the test (see previous section). The standard error of difference gives a reasonable plus or minus on the difference between two scores.
For a high-reliability test (a public examination like the HSC, for instance) two students with equal ability or achievement would score marks up to 3.5 different 68% of the time and more than 7 marks different in the most extreme 5% of cases.
The situation is far worse with less reliable tests as shown in Table 1:
Table 1 Statistics for Different Reliabilities
| Reliability | Standard Error | Standard Error of difference |
For equal students, 5% of the time the difference exceeds |
| 0.96 | 2.5 | 3.5 | 7 |
| 0.90 | 4.0 | 5.6 | 11 |
| 0.84 | 5.0 | 7.1 | 14 |
| 0.77 | 6.0 | 8.5 | 17 |
Assessment marks obtained from class tests or exercises generally have much lower reliability than 0.77, but we tend to average several of them, reducing the standard error. For example, averaging four tasks halves the standard error of measurement (or makes the overall measurement twice as accurate).
These results should make us wary of automatically assuming that a student who gets 52% has 'passed' while a student who gets 48% has 'failed'. They should also make us wary about spending time marking students' work using an overly complicated marking scheme.
Combination of information from several assessment events presents many traps. If we are using a pure criterion-based approach we will have information on whether a student has achieved the criterion on a number of separate assessments. Then we may want to set up an overall criterion for the combination. For instance: the student must reach the acceptable level in all components (or maybe in all but one of the components).
Usually, however, information in the form of marks is combined using a norm referenced approach. Then we must beware of the following situation. If we are combining sets of marks from tests A and B (both as %) in a desired weighting, the actual weighting is determined by the relative standard deviations of the marks from A and B. For example, if a practical assignment with a narrow spread (standard deviation = 5) is combined with an examination with a fairly wide spread (standard deviation = 15) the desired weighting is 50:50, but the actual weighting is 25:75.
The results of combining marks from components of assessment which have different spreads are clearly illustrated in the following example (Clark, 1990). Table 2 shows rankings of students A to L as a result of addition of raw scores.
Table 2: Student Rankings Before Scaling
| Student Assessment Component Total | |||||||||
| A | B | C | D | E | F | G | H | ||
| A | 50 | 50 | 80 | 50 | 25 | 60 | 32 | 42 | 419 |
| B | 95 | 70 | 15 | 80 | 40 | 40 | 30 | 48 | 418 |
| C | 50 | 80 | 65 | 20 | 70 | 75 | 34 | 40 | 414 |
| D | 81 | 60 | 30 | 60 | 60 | 36 | 44 | 42 | 413 |
| E | 94 | 20 | 75 | 50 | 50 | 30 | 47 | 46 | 412 |
| F | 60 | 90 | 20 | 75 | 30 | 45 | 45 | 46 | 411 |
| G | 100 | 40 | 40 | 70 | 20 | 40 | 52 | 48 | 410 |
| H | 10 | 75 | 50 | 55 | 30 | 60 | 56 | 45 | 381 |
| I | 31 | 30 | 60 | 70 | 48 | 35 | 55 | 50 | 379 |
| J | 80 | 10 | 50 | 25 | 30 | 70 | 50 | 55 | 370 |
| K | 20 | 80 | 30 | 30 | 40 | 55 | 60 | 52 | 367 |
| L | 70 | 25 | 10 | 50 | 40 | 50 | 55 | 60 | 360 |
The ranking in Table 3, which is the exact reverse of that in the first, comes from scaling the raw scores before addition.
Table 3: Student Rankings After Scaling
| Student Assessment Component Total | |||||||||
| A | B | C | D | E | F | G | H | ||
| L | 67 | 19 | 0 | 50 | 40 | 50 | 83 | 100 | 409 |
| K | 11 | 88 | 29 | 17 | 40 | 63 | 100 | 60 | 408 |
| J | 78 | 0 | 57 | 8 | 20 | 100 | 67 | 75 | 405 |
| I | 23 | 35 | 71 | 83 | 56 | 13 | 83 | 50 | 404 |
| H | 0 | 81 | 57 | 58 | 20 | 75 | 87 | 25 | 403 |
| G | 100 | 38 | 43 | 83 | 0 | 25 | 73 | 40 | 402 |
| F | 56 | 100 | 14 | 92 | 20 | 38 | 50 | 30 | 400 |
| E | 93 | 13 | 93 | 50 | 60 | 0 | 57 | 30 | 396 |
| D | 79 | 63 | 29 | 67 | 80 | 15 | 47 | 10 | 390 |
| C | 44 | 88 | 79 | 0 | 100 | 63 | 13 | 0 | 387 |
| B | 94 | 75 | 7 | 100 | 40 | 25 | 0 | 40 | 381 |
| A | 78 | 50 | 100 | 50 | 10 | 75 | 7 | 10 | 380 |
The method of scaling the marks is to give the lowest mark zero and the highest mark 100, then determine all other marks by a straight line conversion graph. For more information see Gilmore (1987).
It should be noted, however, that when we inform students that "50% of their marks will be based on classwork and 50% on the exam" we are not necessarily implying that these two areas will contribute equally to their final ranking. Much assessment in Higher Education is not about ranking, but with judging whether students have achieved the objectives of the subject.
This is the process by which the results obtained on assessment tasks which have not been taken in common are made comparable by means of results obtained on a common task. The distribution of marks in each subgroup is adjusted to have the same distribution as the marks achieved by that subgroup on the common assessment task (though usually only the mean and standard deviation are adjusted).
The process is commonly used at the secondary school level. For example, moderation of class assessments over several 2 Unit mathematics classes at a TAFE College by means of a common test, or moderation of assessment marks from all NSW schools by means of the HSC examination.
In the university situation, it may be necessary to carry out a moderation process if a large class is taught in several parallel groups and assessed with different class tests or practicals and a common final examination. However, it seems that the moderation process is rarely attempted at university level.
There are three potential problem areas in moderation. The first is that the calculations need to be done on a computer as they are not usually feasible by hand. Second, the common assessment task should not be measuring something very different from the other tasks. Thus, a written examination is not a valid moderator for practical marks. Third, the common assessment task must truly be in common, i.e. under the same conditions, at the same time, with unbiased supervisors and unbiased marking. If common conditions do not hold, the complete moderation process is invalid.
This is a difficult job, and the details will not be discussed here. It is a more immediate problem in public matriculation examinations such as the HSC. In the university situation, it is used to calculate the weighted average mark (WAM) which is only used for excluding students who perform poorly or awarding prizes to outstanding students.
The most straightforward and the least acceptable method is simply to average the raw percentage marks. This is the method used currently to calculate the WAM at UTS.
A better method is to scale the marks in each subject to the same mean and standard deviation, and then to simply average the marks across all subjects taken (or a certain number of the best subjects taken, as at the HSC). This obviously discriminates against the more intellectually demanding subjects and favours the 'easy options', not something that most teachers would encourage.
Statisticians are generally agreed that the best method of combining marks from different subjects is to use some form of inter-subject scaling to calculate weightings for different subjects before they are averaged. In this process, the marks in each subject are adjusted to reflect the ability of the candidature in that subject. This is the method used at the HSC to obtain a 'Tertiary Entrance Rank'.
Statistically speaking, the WAM as calculated at UTS is a very primitive way of assessing students' overall achievement. However, it may be adequate for picking out the very poorest and the most able students.
Do remember that the aim of statistics is to use the numerical information intelligently. The following quotation illustrates the statistical spirit in assessment, although some people see it as cheating, fudging, or non-statistical:
Before any set of marks or ranks is considered final, the assessments should be examined subjectively by all teachers involved. Any anomalies which appear to have been introduced by the whole assessment process should then be removed by changing the mark or rank of the student(s) concerned. It must be recognised that aggregate marks are not absolute truths: they suffer from errors of measurement, unintended weightings and losses of reliability. They merely provide reference points for later fine adjustment: the final arbiter should be the teacher's professional judgement. It is by incorporating value judgements with weighted assessment aggregates that we develop an evaluation of student achievement. (Secondary Schools Board: "Assessment Guidelines 6")
It is impossible to avoid the subjective element in assessment. Statistics should be used to aid commonsense and professional judgement, not to replace them.
Clark, R., The WAM System. Paper to Interim Academic Board, 90/9, UTS, 1990.
Ebel, R., Measuring Educational Achievement. Prentice-Hall, New Jersey, 1965.
Gilmore, A., Combining Scores (Set, The Best of Assessment). NZCER/ACER, Wellington, NZ and Hawthorn, Victoria. 1987.
Rowntree, D., Assessing Students: How Shall We Know Them. Harper and Row, London, 1977.
Written by Dr Peter Petocz, School of Mathematical
Science and containing a component of a paper to Academic Board
by Dr Roy Clark, School of Teacher Education.
Some of the other key assessment issues not described in the previous sections have been addressed elsewhere.
Assessment procedures to be used by staff at UTS are described in Assessment Procedures Manual which is provided as an attachment to the Report to Academic Board by the Working Party on Assessment, and is available to all staff from the office of the Academic Registrar. Topics covered in that manual include: Responsibilities of Co-ordinating Examiners, Assessors and Examiners; Heads of Schools; Deans; Examination Review Committees; and the Academic Registrar. The attachments cover Recommended Grades for Use at UTS, Interim Results, Council Policy for the Disclosure of Assessment Results, Procedures for Students Requiring Alternative Assessments, and the Academic Registrar's Representative at meetings of Examination Review Committees.
Assessment procedures adopted by both the
University and by academic staff will be subject to changes. The
former as a result of changes to Academic Board resolutions, the
latter following regular reviews as a normal part of a staff member's
curriculum evaluation of their teaching practices. Changes to
University procedures will be notified through updates or supplements
to the Manual. New assessment methods are continually being tried
by members of academic staff. While this booklet contains some
of the range of methods used, it is by no means exhaustive, and
it is not intended that assessment methods be restricted to this
choice or, more importantly, selected from this booklet and subjected
to no further review.
Multiple-choice
Questions
Multiple-choice Questions (MCQs) are a subset of what are referred to as "objective questions". Objective questions are questions which have a correct answer (usually only one). The term "objective" here means there is complete objectivity in marking the test. The construction, specification and writing of the individual questions (items) are influenced by the judgements of examiners as much as in any other test.
The objective test is largely used to test factual material and the understanding of concepts. Because of the objectivity and ease of marking it is frequently used for testing large groups. It is claimed that skilled items writers can develop items to test higher level intellectual skills (Cannon and Newble: 1983) but if the perception of students is that these types of questions usually test the recall of facts, then they will prepare for them accordingly.
It is to be noted that the quality of an objective test is determined by the skill of the constructors of the test.
While the following notes refer specifically to the most common form of objective tests (MCQs), many of the comments are relate to all objective questions. Some of the advantages and disadvantages of the range of questions is given in Table 4 on page 24.
Multiple-choice Questions are probably the most widely used of objective tests. Such questions are normally composed of four parts:
STEM - question or incomplete statement
OPTIONS - suggested answers or completions
DISTRACTERS - incorrect responses
KEY - correct response
While the stem is, in most cases, written material, other material such as graphs, diagrams, sets of results may be used. In these cases it is probable that abilities other than recall may be tested. In some cases a number of questions (items) may be related to the same material. In these circumstances, care must be taken to make the questions independent (i.e. the answer to one question not depending on an answer to a previous one).
In some cases, MCQs may be used to test higher abilities by asking students to judge the most appropriate answers to a problem from several "correct" ones. Obviously, great care is required for the setting of such a question.
It is usual to have four or five options, with five options giving the most reliable test. However, it may be difficult to provide five plausible options and in this case it is better to stay with four.
Dunstan (1971), in a short monograph on
multiple choice testing, summarises the steps involved in writing
MCQs. He makes three important general statements which briefly
may be summarised as follows:-
More detailed setting guidelines and considerations
are given below.
1. Objectives:
The answers to these questions are important. Because of the increasing use of computer marked tests, MCQs are becoming increasingly popular, particularly where large classes are involved.
In some cases, the use of MCQs may not be appropriate. For example, it may be difficult to find suitable distracters for a question. In other words, the material and abilities under test should determine the nature of the examination rather than fitting the examinable materials to a specific type of test.
2. Stem:
It is most important that the stem is clearly
and unambiguously worded. The language used should be simple and
clear. If these guidelines are not followed, the test item becomes
one in which comprehension is tested as much as the material on
which the question is based. The stem should also state clearly
what is expected of the student before the options are read. This
requirement would rule out use of the question "which of
the following is correct?'
3. Options
Options should be independent of one another, consistent in logic and grammar to the stem and the language used must be clear and simple. In particular, the examiner should avoid clues to the correct answer and clues which will allow a student to discard distracters even if little is known of the material under test. Poorly constructed items may allow a student to reduce the number of plausible distracters and hence increase the chance of guessing the correct answer. In particular, the person constructing the test should AVOID USING:
One of the advantages of using multiple-choice questions is the statistics on the tests which are easily obtainable (particularly when the tests are computer scored). The parameters which define the quality of the test item are discrimination (D) and facility (F).
Discrimination compares the number of correct responses to an item for the upper and lower 27% of the class (based on the total test score). If for any one item the number of correct responses from people who are in the lower 27% for the whole test is greater than the number from those in the top 27%, then the item may not be effectively discriminating between students.
Facility (F) is the percentage of the class obtaining the correct answer. In general -
if F < 30% the question is hard
if F = 30-75% the question is satisfactory
if F > 75% the question is easy
Whether or not items are acceptable in terms of difficulty is largely the judgement of the examiner. A hard item with a high discrimination may be useful in establishing a rank order of students. An easy item with a low discrimination is probably a give-away question and consideration might be given to using another form of assessment. Items with acceptable discrimination and facility may be stored for future use. In this way an item bank may be built up over time and questions selected randomly for tests.
The quality of the test as a whole is measured by its validity (v) and reliability (r). Validity for lecturer-developed tests is determined in the light of course content and course objectives to determine the degree to which the test, content and objectives coincide.
Reliability is, in effect, the probability that the same score on a test would be achieved, on repetition of the test, by the same or an equivalent group of students or when the test is scored by another examiner. Values of this statistic range from 0 to 1 and a value of 0.7 or greater is generally acceptable. Objective tests in general have a higher reliability than essays primarily because of marker objectivity. Reliability increases with length of test and if questions with high discrimination values are included. Clear statements of instructions and items will also increase reliability.
A note of caution: high reliability does not imply high validity. An examiner could consistently be testing other objectives than those believed to be under examination.
Objective tests are often criticised because they encouraging guessing. Marking schemes which can take account of this assumption include:-
(a) Raising of the pass mark: In a test of 100 questions with 5 options per question, random guessing should allow a score, on average, of 20.
Hence the range of marks to be considered is from 20 to 100. The mid-point of this range (or pass mark) is 60, rather than 50.
This method assumes that random guessing takes place and that a score of 60 represents a satisfactory attainment of course objectives. Both these assumptions may or may not be valid.
(b) Deductions for guessing: A score of 1/(n-1) is deducted for each incorrect answer where n is the number of options per item. There is no penalty for an omitted answer. Assumptions are made when this method is applied. Firstly, it is assumed that incorrect answers are made on the basis of random guessing and that omissions are made only on the basis of insufficient information. Secondly, it is assumed that all options are equally attractive. Other considerations which may be taken into account when guessing corrections are used include:
(i) the influence of personality and test-taking strategy on the scores of students - confident students may guess; cautious students may adopt "error avoidance" schemes;
(ii) the overflow of scoring to subtract from other areas, which means that the score does not vary directly with knowledge.
Obviously, the quality of the item writing will have a large bearing on the way students select options as being correct or otherwise. Where poor distracters are used students will be able to use logical deductions to assist them in their search for the correct answer rather than guessing. In this case one of the assumptions made in the use of guessing corrections does not apply.
Billing, D.E. (1973) Objective Tests in Tertiary Science Courses in D.E. Billing and B.F. Furniss (Ed.) Aims Methods and Assessment in Advanced Science Education, Heydon & Son, London, p.131-148
Billington, D.R. (1981) The uses and abuses of assessment in biochemistry education in C.F.A. Bryce (Ed.) Biochemical Education, Croom Helm, London, p.95-122.
Cannon, R.A. and Newble, D. (1983) A Handbook for Clinical Teachers, Lancaster, MTP Boston: p 97-105.
Dunstan, M. (1971) A Guide to the Planning Writing and Review of Multiple Choice Tests, Tertiary Education Research Centre, University of New South Wales
Gibbs, G., Habeshaw, S. and Habeshaw, T, (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol.
Gronlund, N.E. (1982) Constructing Achievement Tests, 3rd. Edition, Prentice Hall, p.36-70
Hudson, B. (1973) Assessment Techniques - An Introduction, Methuen, London, p.122-126
Lennox, B. (1974) Hints on Setting and Evaluation of Multiple Choice Questions of the One from Five Type, Association for the Study of Medical Education, Dundee, Scotland
Stratton, J.J. (1981) Recurrent Faults in Objective Test Items, Teaching at a Distance, 20, p.66-73
Education Research and Development Unit, Queensland University of Technology.
Table 4: Advantages and Disadvantages of Different Types of Tests
| Item Type | Advantages | Disadvantages |
| Multiple choice | Simplicity in structure and make changes of guessing; students not exposed to any false statements; applicable to branched learning programmes and other feedback | Often impossible to think of four or five possible detractors; sometimes two or more alternatives may be correct, depending on the sophistication of the students |
| True/false | Simplicity; some statements can only be written with two rather than five alternatives | Too easy to guess, so this must be penalised; few areas are so clear-cut that there is no alternative besides 'true' and 'false'; students exposed to false statements |
| Multiple completion | Easy to construct since several correct answers are permitted; few incorrect distractors need to be constructed; no false statements | Instructions to students are complicated |
| Situation | Comprehension, application and evaluation can be easily tested; relationships between areas can be made by connecting items to a common situation | Difficult to think of situations which are not too similar to those the students has encountered, and yet which s/he has knowledge and ability to answer; complex to construct |
| Matching | Comprehension can be easily tested; related areas can be brought together; few distractors per item since one list services several items | Complex to construct; instructions are complicated; all responses must be homogeneous in being related in some way; gives false impressions of clear distinctions between categories (eg. of material, properties, interactions) |
| Assertion/Reason | Reasoning ability is easily tested; large numbers of distractors unnecessary | Students exposed to false statements; instructions are complicated; difficult to ensure that a 'reason' is either completely false at all levels of sophistication or is uniquely correct. |
Standard forms of essays require students to
Discuss a quotation, or
Write an essay on ........., or
Describe, Give an account of, Compare, Contrast, Explain ......., or
Assess, Analyse, Evaluate .........
While these types of questions give students the freedom to choose what they will concentrate on and to structure their work themselves, they may also leave the weaker students in some dilemma as to what is required. In addition to these types of questions there are a range of alternatives which can be employed to fulfill certain roles or suit different objectives. Three of these are briefly outlined below. These, and other variations appear in Gibbs et al (1986).
In role play essays, students respond to an essay question from the perspective of a position given in the essay question.
Role play essays help students see the relevance of the task and take an interest in it. Their writing often becomes more fluent and natural. Even small elements of simulation or role play can dramatically change students' approach to questions.
There is a danger of encouraging too flippant an approach, but this can be kept in check by careful phrasing of the question. For example ask students to write to someone in an official position, such as the Minister for Higher Education, or a superior, such as their Managing Director.
Write a letter to the Minister for Higher Education protesting about the lack of university places in Australia, giving economic arguments and emphasising evidence in Government Reports.
Advise Weybridge Electrical Ltd. (by whom you have been employed as a consultant) on the suitability of the circuit designs in Appendix I given the performance specifications in Appendix II.
Structured essays, require students to respond to an essay question which contains specific areas or parts of the questions which require an answer. For example:
Undertake a stylistic analysis of the following passage. Select, arrange and comment on features of syntax, lexis, semantics and (where relevant) phonology. Relate the artistic effects of the passage to the writer's choice of language.
By specifying the content required in an essay it is possible, when marking, to be clearer whether students know about and understand the specific things which you think matter.
It is difficult to know whether students would know which things matter without prompting. However, this type of essay is useful when you are testing specific knowledge and techniques.
Identify and discuss some of the determinants of urban land values and their impact on urban development. In your answer you should:
a) define the following terms; property rights in land; zoning; site value rating,
b) explain the influence of these terms in determining land values,
c) select one activity of Local Government and one market factor which affects market values and explain how each might influence urban development.
Students are supplied with data or evidence. Using that evidence (which in many subjects with mini-projects or laboratory exercises students may have collected themselves) students are asked to write an essay in which they address a question on that evidence.
The question and the data can relate directly to an exercise previously conducted by the students in which they collected, analysed and interpreted data. Interpretation questions require the students to undertake the analysis "live" and this can avoid regurgitation.
You own a house in a developing suburban area but are considering selling your property and moving closer to the city centre. Given the following demographic data
. . . . . . . . .
. . . . . . . . .
What economic and social factors would you consider in coming to a decision?
Grading of essays is a notoriously unreliable activity. If we read an essay at two different times, the chances are good that we will give the essay a different grade each time. If two or more of us read the essay, our grades will likely differ, often dramatically so. We all like to think we are exceptions, but study after study of well meaning and conscientious teachers shows that essay grading is unreliable (Ebel, 1972; Hills, 1976; McKeachie, 1986; White, 1985). Eliminating the problem is unlikely, but we can take steps to improve grading reliability.
First, using a scoring guide helps control the shifting of standards that inevitably take place as we read a collection of essays and papers. The two most common forms of scoring guides reflect the two approaches to grading most widely used in universities: analytic and holistic.
Those who use analytic scoring guides identify important components of the essay and assign marks to each component. As they read the essay, they award marks up to the limit specified by the scoring guide and then total the points to determine the essay's grade. An analytic scoring guide is included as the first example at the end of this section. A variation on the analytic method used in the subject Securities Market Regulation in UTS's School of Finance and Economics, is included as the second example.
Holistic grading methods assume that an essay is other than a sum of particular parts so we read the essay as a whole. Whereas the analytic scoring guide designated marks for particular aspects of the essay, the holistic scoring guide describes the characteristics of excellent, good and not-so-good essays.
An example of a holistic scoring guide is attached as the last example.
Gibbs, G., Habeshaw, S. and Habeshaw, T, (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol, pp. 11-26.
Erickson, B.L. and Strommer, D.W. (1991). Teaching College Freshmen. Oxford: Jossey-Bass, pp 145-148.
Ebel, R.L. (1972). Essentials of Education Measurement. Englewood Cliffs, N.J.: Prentice-Hall
Hills, J.R. (1976). Measurement and Evaluation in the Classroom. Westerville, Ohio: Merrill.
McKeachie, W.J. (1986). Teaching Tips: A guide for the beginning teacher. Lexington, Mass.: Heath.
White, E.M. (1985). Teaching and Assessing Writing: Recent Advances in Understanding,Evaluating and Improving Student Performance. San Francisco: Jossey-Bass.
Analytic Essay Scoring Guide (Erickson and Strommer, 1991)
Total marks possible: 6
Statement of position: 1 mark
The essay clearly states the students' position. One does not have to read between the lines.
Support for the position: 2 marks
The essay cites examples or evidence in support of the position. The quality or persuasiveness of the evidence is worth one mark. Originality is worth one mark.
Statement of an alternative position: 1 mark
The essay raises a reasonably significant objection, counterargument, or alternative to the position taken.
Refutation of the alternative: 2 marks
The essay provides examples or other evidence that render the alternatives false or less persuasive.
| GRADING OF FINAL EXAM The final exam is open book and relates to understanding the process, players, important factors and alternative solutions affecting a specific regulatory issue. Your assessment is based on your ability to effectively argue a position regarding a specific regulatory issue using the tools and information developed in the course. |
|||||
Michael and Mark have read your final exam paper. We have awarded you a mark of _____ out of twenty. This follows because we believe that your essay achieved: |
|||||
| all the major and minor objectives of the question | |||||
| all the major objectives of the question, but some of the minor ones were not | |||||
| all the major objectives of the question, but many of the minor ones were not | |||||
| most of the major objectives of the question, but most of the minor ones were not | |||||
| some of the major objectives of the question, but none of the minor ones were | |||||
| only a few of the major or minor objectives of the question | |||||
| none of the major or minor objectives of the question | |||||
This result has come from our assessment of your paper in terms of a number of categories of achievement. The ticks below indicate where you stand with regard to each set of statements. |
|||||
| D | C | P | F | ||
| STRUCTURE | |||||
| Essay relevant to topic | Essay has little relevance | ||||
| Topic covered in depth | Superficial treatment of topic | ||||
| ARGUMENT | |||||
| Logically developed argument | Essay rambles and lacks continuity | ||||
| Accurate presentation of factors | Much questionable or inaccurate evidence | ||||
| Rigorous critique of key concepts | Lack of demonstration of key concepts | ||||
| ORIGINALITY | |||||
| Original and creative thought | Little evidence of originality | ||||
| STYLE | |||||
| Fluent piece of writing | Clumsily written | ||||
| Succinct writing | Unnecessarily repetitive | ||||
| PRESENTATION | |||||
| Legible and well set out work | Untidy and difficult to read | ||||
| Reasonable length | Too long/short | ||||
| MECHANICS | |||||
| Sentences grammatical | Many ungrammatical sentences | ||||
| Effective use of figures/tables | Figures/tables add little to argument | ||||
| Correct spelling throughout | Much incorrect spelling | ||||
| SOURCES | |||||
| Adequate acknowledgement of sources | Some plagiarism | ||||
Adapted from Educational Services and Teaching Resources, Murdoch University. |
|||||
Highest possible score: 6
6: The essay clearly states a position,
provides support for the position, raises a counterargument or
objection, and refutes it. The evidence, both in support of the
position and in refutation of counterpositions, is persuasive
and original (that is, drawn from the student's own observations,
not borrowed). The essay tackles a significant objection or counterargument,
not a trivial one. The relationships between position, evidence,
counterargument, and refutation are clear, and the essay does
not contain extraneous or irrelevant information.
5: The essay states a position, supports it, raises an objection
or counterargument, and refutes it. The essay may, however, contain
one or more of the following ragged edges: evidence is not uniformly
persuasive or original; the counter-argument is not a very
serious threat to the position; one has to read between the lines
to see relationships between ideas and some ideas seem out of
place or irrelevant.
4: The essay states a position and raises a counterargument, but
their is well developed. The objection or counterargument considered
may lean toward the trivial. The essay may also seem disorganised.
Nonetheless, the essay should receive a 4 in acknowledgement of
the cognitive complexity of the task. It is more difficult
to address arguments and counterarguments than it is simply to
support one line of argument.
3: The essay states a position, provides strong and original evidence
supporting the position, and is well organised. However, the essay
does not address possible objections or counterarguments. Thus,
even though the support seems stronger and the essay
may be more well organised than the 4 essay, it should not receive
more than a 3.
2: The essay states a position and provides some support, but
it doesn't do it very well. Evidence is scanty, general, trivial
or not original. The essay achieves its length largely through
repetition of ideas and inclusion of irrelevant information. The
overall impression is that the essay has been dashed off at the
last minute.
1: The essay does not state the student's position on the issue.
Instead, it restates the position presented in the assignment
and summarizes the evidence discussed in the text or in class.
The essay may include an occasional "I agree with,"
but it provides nothing beyond what was said in class or in the
readings. The essay receives a 1 rather than a 0 because there
may be some merit to being able to summarise what the author of
the text said.
The following checklist, devised as a guide to helping students learn from their assignments is from Gibbs, G.and Habeshaw, T. (1989). Preparing to Teach. Bristol: Technical and Educational Services, p 125.
Commenting on assignments takes skill: it
is not just a matter of crossing out mistakes. Use this checklist
to revue the way you comment on your students' work.
Tick if you ...
| start off with a positive, encouraging comment | |
| write a brief summary of your view of the assignment | |
| balance negative with positive comments | |
| turn all criticism into positive suggestions | |
| take general suggestions for how to go about the next assignment | |
| ask questions which encourage reflection about the work | |
| use informal, conversational language | |
| explain all your comments | |
| suggest follow-up work and references | |
| suggest specific ways to improve the assignment | |
| explain the mark or grade, and why it is not better (or worse!) | |
| offer help with specific problems | |
| offer the opportunity to discuss the assignment, and your comments |
Short
Answer Questions
A large proportion of assessment
items make use of short answer questions of some form (in assignments,
quizzes, examinations, laboratory tests). These questions vary
in expected student response from one word or several lines to
over a page, and include forms such as complete the sentence,
supply the missing line, problems and exercises in science-based
subjects, short descriptive or qualitative answers, essay plans,
diagrams with explanation, etc. The diversity of form means that
no generic description is possible, but they are included in this
section for completeness.
Make the questions precise
Direct questions are better than incomplete statements.
If a numerical answer is required, indicate the units and degree of precision required.
Prepare a structured marking sheet
Allocate marks or part-marks for acceptable answer(s).
Be prepared to accept other equally acceptable answers, some of which you may not have predicted.
Mark questions with the following points in mind
Mark anonymously.
Have different markers for different sets of questions.
For a checklist to help students learn from assignments, see page 31
One example of a "long" short-answer question in chemistry, and detailed marking scheme is included.
| This question caries 25 percent of the marks
for this assignment and tests Objectives 1 and 9 of the
course. The concept of ionic radius is used extensively in chemistry. What observations lead one to suppose that the concept of ionic radius has any validity? Indicate how values of ionic radii may be obtained. Your answer should also indicate potential problems in the use of ionic radii. Marking Scheme: It is quite acceptable for this question to be answered in note form. You may award full marks even if every point is not included, provided that you feel that the answer is well constructed. |
|
| Marks | |
| Interionic distances in crystals obtained from x-ray diffraction | 3 |
| Impossible to say where one ion ends and the other begins | 3 |
| Series of compounds indicates constant(ish) factor when cation (or anion) is changed. Here values should be quoted | 6 |
| Appears that ionic rad assignable, are roughly additive | 3 |
| Assignment of ionic radii should refer to the Lande method with 'touching' I- in Lil | 6 |
| Problems Additivity not strictly applicable. Values vary according to assignment method. Values may vary with environment (eg co-ordination number) |
4 |
Newble, D. and Cannon, R. (1989). A Handbook for Teachers in University and Colleges. New York: Kogan Page, p 107.
Inorganic Chemistry: Concepts and Case Studies,
(1981). Milton Keynes: Open University Press
Conventional examinations which are normally unseen by students prior to the day they are attempted are widely used, partly because they have been used for years without question, and partly because they appear to have some advantages. These advantages include being able to sample a range of student learning and testing the students own work. However they also have many disadvantages, some of which can be overcome by selecting alternative examination methods.
The main disadvantage of unseen examinations is the extent to which they encourage students to memorise information for examinations rather than attempting to understand it as a component of their overall course. Open book exams go some way towards remedying this disadvantage. Two alternative methods of examining student understanding which remove the incentive to memorise are outlined below. In both cases, the suggestions work best with questions that do not have a unique correct answer.
An examination method more in keeping with the type of "tests" students will encounter in their careers is the take home exam. Students are given the examination paper and 2 - 7 days to submit their responses. It assesses their ability to research, redraft, and use resources, and places less emphasis on speed and memory than conventional exams.
An excellent way to inform students about what really matters in your subject is to show them the examination paper at the beginning of the semester. This examination paper should contain a broad question on each major topic of the subject (say eight questions), and students should be told that the final examination will only contain (say) three of them and that they will be required to answer all three.
Advantages
Disadvantages
Examples
A variation on this idea is employed in the subject Law for Marketing Managers, offered through the Faculty of Law for Business students. In that subject, the lecturer hands out 12 to 13 questions early in the semester with the advice to students that the final examination will contain 4-5 questions made up of parts or a whole of, and variations on, these 12-13 questions. Using the full range of resources available to them (library books, cases, etc) students prepare their answers free of examination stresses, and are able to take those answers with them into the examination room.
In all assessment schemes it is important that students are given clear criteria at the beginning of the semester and that these criteria do not change during the semester.
Gibbs, G., Habeshaw, S. and Habeshaw, T, (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol, pp. 49-64.
James Cooper, Faculty of Law.
A learning contract is a structured method whereby each student, in consultation with a staff advisor, designs and implements manageable learning activities. The emphasis is on making each activity relevant to those professional and personal needs of the student which are consistent with the aims of the course and/or subject.
Advantages
The principal advantages of the learning contract method are:
A learning contract is an agreement between a student and a staff adviser that the student will undertake a specified activity and produce evidence that the activity was successfully completed.
The role of the adviser is to negotiate the contract, monitor progress and to participate in the final assessment. Normally guidelines are developed explaining how to set out the contract, the parameters within which the contract should be negotiated, and any generic assessment criteria.
Typically a "Learning Contract Proposal' is developed which comprises the following elements (see the examples at the end of the section):
Purpose
What is required here is a clear statement of the purpose of the contract. This may be stated as an objective, a problem, an issue etc. The purpose will need to be appropriate to the subject or course and be sufficiently challenging to warrant inclusion in a degree level program.
Strategies and resources
Here is indicated the references students intend to consult, people they will arrange to interview, research they conduct, material they prepare to gather data (e.g. evaluation form), places they will visit etc.
What will be produced
Mostly students produce written work of some kind, e.g. a report, an essay, a journal, a book review etc. But other forms of presentation are also acceptable, e.g. video-recording, audio-recording, charts and diagrams, models, etc.
Adviser comments
Here the adviser will make comments and suggestions on the contract proposal. The adviser will draw attention to any weaknesses in the proposal, note any special conditions, and set out the assessment criteria.
1. The learning contract method may be linked to a particular subject or it can be a subject in its own right (e.g. referred to as Individualised Projects). In the latter case, students have more degrees of freedom in developing their contracts, but the expertise of advisers is stretched. Both approaches have merit and may be used concurrently in a given course, but they need to be administered differently.
2. The learning contract can be thought of as an assessment device or a learning device. This has implications for staffing allocations. If it is merely an assessment device connected with a particular subject, then the lecturer will receive no additional teaching hours beyond the face-to-face hours allocated. If it is seen as a learning device, with the adviser playing a crucial and on-going role, then teaching hours need to be allocated to the activity. In this way it can be seen as an alternative to learning in a large group. Instead, there is face-to-face, one-on-one or small group contact with an adviser.
3. The assessment criteria may be generic or individually negotiated. In the latter case, different criteria may be applied to the same subject, which certainly raises equity considerations. These may be overcome by having a generic standard which is applied to the criteria developed for individual contracts.
4. It is necessary to ensure that learning contract work has not been submitted elsewhere for credit. Some kind of statement on the cover of the contract may be required, but there is always the marginal case where some other work has been adapted or edited. It is normal practice to keep records of contracts completed.
5. The learning contract, when completed, may deviate significantly from the proposal, but is otherwise satisfactory. Ideally, contract proposals should be re-negotiated; notwithstanding this, if the final product fits within the parameters it should be assessed in the normal way.
6. Some contracts may be negotiated which are not within the adviser's area of expertise. This will invariably occur with learning contracts. The problem is that the level of advice which can be given is less than would normally be the case. Also the adviser is not in a good position to assess the final product. Instances such as these can be handled by re-allocating advisers for a given contract, or using a back-up marker, or even using a panel of advisers to check on marking procedures.
Caffarella, R. and Caffarella, E. (1986). Self-directedness and learning contracts in adult education. Adult Education Quarterly, 36, 226-234
Hiemstra, R and Sisco, B. (1990). Individualising instruction. San Francisco: Jossey-Bass.
Knowles, M. (1986). Using learning contracts. San Francisco: Jossey-Bass.
O'Donnell, J. and Caffarella, R. (1991). Learning contracts. In Michael Goldsmith (Ed.) Adult learning methods. Malabar: Krieger. (pp133-160)
Tompkins, C and McGraw, M.J. (1988). The negotiated learning contract. In David Boud (Ed.) Developing Student Autonomy in Learning. London: Kogan Page.
School of Adult and Language Education
School of Teacher Education
School of Adult Vocational Education
School of Nursing Health Studies
School of Information Studies.
Professor David Boud, School of Adult and Language Education
Associate Professor Mark Tennant, School of Adult and Language Education
Trish Farrar, School of Nursing Health Studies
| UNIVERSITY OF TECHNOLOGY SYDNEY, KURING-GAI
CAMPUS NURSING STUDIES VIA GUIDELINES FOR COMPILING A PERSONAL RESOURCE FILE In your Personal Resource File, you should include the following: |
|
| 1. | journal articles to which you have referred in the "Learning Procedures" column of your learning contract; |
| 2. | written information which helps you to meet your learning objectives eg. pharmaceutical information, equipment instructions, newspaper cuttings etc; |
| 3. | completed worksheets which provide evidence of accomplishment of your learning objectives; |
| 4. | nursing care plans which you completed during your nursing practice experience placement; |
| 5. | rating scales which you may have devised to demonstrate your learning progress. |
You should acknowledge the information contained in your Personal Resource File in your Reflective Journal, in accordance with the "Guidelines". A suggested method for presenting your Reflective Journal and Personal Resource File is to use a ring-binder and plastic sleeves. The Reflective Journal can be included at the front, with your Personal Resource File enclosed in the plastic sleeves. Each plastic sleeve could contain the information pertinent to one learning objective. Don't forget to include your Learning Contract, as this guides your Reflective Journal and Personal Resource File. The above is only one suggested method of presentation - it is not mandatory. |
|
|
Student's Name: Agency/Facility: Facilitator: Date Negotiated: |
|||
| LEARNING OBJECTIVES "What am I going to learn?" |
LEARNING RESOURCES AND STRATEGIES "How am I going to learn it?" |
EVIDENCE OF ACCOMPLISHMENT "How am I going to know that I have learned it?" |
CRITERIA AND MEANS OF VALIDATING EVIDENCE "How am I going to prove that I have learned it?" |
| 1. To update my knowledge of cardiac pharmacy. |
- Read journal articles in "Current Therapeutics" and "Heart and Lung". - Collect drug literature from medications I administer. |
- Summarise articles for annoated bibliography. - Complete inventory of drugs which I administer in CCU. |
Present annotated bibliography
in Personal Resource File. Include inventory in Personal Resource File. |
| 2. To become proficient in haemodynamic monitoring. |
- Read articles in "Nursing
'89". - Consult with the Clinical Nurse Specialist |
- Complete the exercises and
worksheet on haemodynamic monitoring in "Critical Care Nursing" Work Manual. - Care for a patient with haemodynamic monitoring without feeling apprehensive. |
Ask the Clinical Nurse Specialist
to supervise and assess me during haemodynamic monitoring. - Include completed worksheet in Personal Resource File. - Document my personal achievement in haemodynamic monitoring. - Reveal my feelings about haemodynamic monitoring in my Reflective Journal. |
| 3. To develop a new rotating roster in the Coronary Care Unit. |
- Consult with Time Management consultant - Research the area on Med Line |
- Write an annotated bibliography
of sources consulted. - Explain the new system to my colleagues and debate the pros and cons. |
- Implement the new roster system. - Include a summary in my Personal Resource File. - Present "spotlight". |
Peer assessment, in which students comment on and judge their colleagues work, has a vital role to play in formative assessment, but it can also be used as a component in a summative assessment package.
One of the desirable outcomes of education should be an increased ability in the learner to make independent judgements of their own and others' work. Peer and self assessment (See page 45) exercises are seen as means by which these general skills can be developed and practised. A peer rating format can encourage a greater sense of involvement and responsibility, establish a clearer framework and promote excellence, direct attention to skills and learning and provide increased feedback (Weaver and Cotrell, 1986).
In terms of summative assessment, studies have found student ratings of their colleagues to be both reliable and valid. Orpen (1982) found no difference between lecturer and student ratings of assignments in terms of average ratings, variations in ratings, agreement in ratings or relationship between ratings. Arnold et al. (1981) reported that peer ratings of medical students were internally consistent, unbiased and valid. Other studies suggest there is variation according to factors such as age of the student (Falchikov, 1986).
Reports of the types of assessment where peer assessment is used for summative purposes include essay writing, clinical skills, speeches and oral presentations, architectural designs, interpersonal skills, photography and small group activities (Kane and Crawford, 1989). In all cases, the contribution to the overall assessment result is small (10-30%).
The second of the two examples of Peer Assessment forms attached is used to determine an individual's contribution to a group's activity. For more details see Group Work, page 72.
Advantages
Disadvantages
Arnold, L. et al. (1981). Use of peer evaluation in the assessment of medical students. Journal of Medical Education, 56, 35-42.
Falchikov, N. (1986). Product comparisons and process benefits of peer group and self assessments. Assessment and Evaluation in Higher Education, 11, 146-166.
Orpen, C. (1982). Student versus lecturer assessment of learning: a research note. Higher Education, 11, 567-572.
Weaver, W. and Cotrell, H.W. (1986). Peer evaluation: a case study. Innovative Higher Education, 11, 25-39.
Extracts and examples from: Kane, R.L. and Crawford, J. (1989). Peer ratings of oral presentations by students: some preliminary data. UTS Faculty of Business Working Paper.
| PROJECT PRESENTATIONS PEER RATING FORM This form would be used to RATE THE PROJECT PRESENTATION YOU WILL BE HEARING. Please try to rate honestly. The student giving the presentation will not see this form, but will receive a tally of all the responses to their presentation. For all categories, please CIRCLE the number that isthe nearest approximation to your opinion. Please do not circle more than one response for each item, or place marking between the numbers on the scale. IN YOUR OPINION, HOW EFFECTIVE WAS THIS PREENTATION IN TERMS OF: |
|||||
| Weak | Fair | Good | Very Good |
||
| 1. | Providing a brief summary of the project (purpose, methods,results, conclusions, etc)? | 1 | 2 | 3 | 4 |
| 2. | Explaining and illustrating the important points? | 1 | 2 | 3 | 4 |
| 3. | The knowledge shown by the presenter? | 1 | 2 | 3 | 4 |
| 4. | Provoking and cntrolling discussion? | 1 | 2 | 3 | 4 |
| 5. | The extent to which the issues were new to you? | 1 | 2 | 3 | 4 |
| 6. | Presentation style? | 1 | 2 | 3 | 4 |
| 7. | Timing? | 1 | 2 | 3 | 4 |
| 8. | Gaining and holding your interest? | 1 | 2 | 3 | 4 |
| 9. | The extent to which you learned something from it? | 1 | 2 | 3 | 4 |
| 10. | The likely usefulness of this learning to you? | 1 | 2 | 3 | 4 |
|
2 Inadeaquate 3 4 5 Passable 6 Reasonably good 7 Good 8 Very good 9 Outstanding in every respect 10 Perfect in every respect |
|||||
Examples of Peer Rating Forms (Securities Markets Regulation)
PEER EVALUATION Name ____________________________________ Group __________________________ |
Please try to assign scores that reflect how you really feel about the extent to which the other members of your group contributed to your learning and/or/your group's performance. This will be your only opportunity to reward the members of your group who actually worked hard on your behalf. If you give everyone pretty much the same score you will be hurting those who did the most and helping those who did the least. Instructions: In the space below please rate each of the other members of your group. Each member's peer evaluation score will be the average of the points they receive after the highest and lowest scores have been deleted and the scores have been standardised so that the average peer evaluation score for all groups is identical. To complete the evaluation you should: 1. List the name of each of the members of your group in the alphabetical order of their last names. 2. Assign an average of ten points to the other members of your group. (Thus, for example, you should assign a total of 50 points in a six member group; 60 points in a seven member group; etc.) 3. Differentiate some in your ratings, eg. you must give at least one score of 11 or higher (maximum = 15) and one score of 9 or lower. |
|
2. ______________________________ __________ 3. ______________________________ __________ 4. ______________________________ __________ 5. ______________________________ __________ 6. ______________________________ __________ 7. ______________________________ __________ 8. ______________________________ __________ |
Additional Feedback: In the space below would you also briefly describe your reasons for your highest and lowest ratings. These comments - but not information about who provided them - will be used to provide feedback to students who would like to receive it. |
| 1. Reason(s) for your highest rating(s). (Use back if necessary) |
| 2. Reason(s) for your lowest rating(s). (Use back if nescessary) |
The following commentary is supplied by a user of oral exams in the School of Teacher Education at UTS
By the fifth semester of the sequence students have had considerable experience in thinking about literature and in articulating their responses to it. An oral examination at this stage does not make demands upon them which they feel incapable of meeting.
By this stage some students have developed particular areas of interest and are beginning to become independent thinkers about literature. The oral examination gives them an opportunity to take up points and pursue them in a way not always appropriate in a response to a set examination or essay question. (It is always possible to say to a student, in an oral examination: "That is an interesting idea. Would you like to develop it further.")
The oral examination is more demanding because it allows the examiner to probe understanding. Students cannot hide behind a learned opinion or borrowed position. (It is always possible to ask: "Just what exactly do you mean by that? Why do you think that critic says that? What about the position taken by X?")
The oral examination requires students to think on their feet, i.e. to adapt their knowledge and insights to a variety of demands and defend their position. Students in the final semesters of their sequence should be able to perform in this way. At the beginning of their course of study, on the other hand, it is probably sufficient to expect that students will arrive at some understanding and be able to express it.
Each student has a session which lasts approximately twenty minutes.
Students have selected a general topic on which they wish to be examined. e.g. King Lear. Every student who has selected that topic will be asked the same set of questions.
The prepared questions demand a hierarchy of responses. Some are designed simply to establish that the student has a basic, literal knowledge of the topic (e.g. of the events in King Lear). Some are designed to see whether the student has thought about the significance of that knowledge (e.g. "In the source play for King Lear Cordelia does not die. Why do you think Shakespeare modified the story to include her death? How do you see her death in relation to the 'meanings' of the play as you understand them?). Some are designed to encourage students to pursue any particular interests or insights they may have had ("Do you agree with...?" "If you were producing a performance of the play would you...?") and there is always a final question: "Is there anything else you would like to say about this play/topic that we haven't covered?" Some questions are designed to see whether students have related their thoughts about the topic to wider issues (e.g. "King Lear was one of the least often performed of Shakespeare's plays during the nineteenth century. Since World War II it has been one of the most frequently performed and discussed plays. Have you any thoughts about why this might be so?").
If students have difficulty in answering questions some prompt questions are asked. But if the difficulty persists, such that it is apparent that the student does not have sufficient knowledge or understanding to answer the question, we pass on to the next question. It is important, in that situation, to move on without undermining the student's confidence by making it apparent that they have 'failed' a section of the examination.
Ideally two or three staff members conduct the exam, sharing the asking of questions, and discussing the student's responses at the conclusion of the examination. In recent years staffing constraints have made this difficult, and sometimes it has not been possible to use additional staff. For this reason the oral examination is never worth more than 40% of the total. Notes are made on the student's responses as the examination proceeds, and a grade is decided before the next student is dealt with.
Broad grades rather than precise marks are awarded, but perhaps with a plus or minus modification, which may be taken into account when determining the final, overall result for the subject.
For a pass grade a student needs to demonstrate a basic knowledge of the topic and some considered, but not necessarily subtle or original, views about its significance. For a credit grade a student would need to demonstrate, as well as the above, some understanding of the range of possible views about the topic and be able to defend the personal position which (s)he has adopted. For a distinction grade a student would need to display some understanding of the more subtle issues involved in the topic, and demonstrate some critical thinking about these issues. For a high distinction grade a student would need to demonstrate a superior understanding of the issues involved and some original thinking about them.
The oral examination is an excellent discriminator. It is readily apparent if students do not have a sound basic understanding of their topic; prepared answers are no defence. Students who have real interest and genuine insights are able to develop these effectively in this context, because the range of questions does not limit them to exploring only one issue or set of related issues.
The oral examination is an opportunity for students to use their oral communication skills in a formal but flexible context.
Although the examination takes a long time to administer there is, of course, minimal 'marking' time required. The size of the group involved has never exceeded thirty students; it would be very difficult, with limited staff, to use this method with large groups.
Surveys of student opinion have elicited favourable comments on the oral examination as a method of assessment. Most students say they found the experience much less threatening than they anticipated, and felt that it gave them an opportunity to demonstrate what they had learned during the semester.
Margery Hourihan, School of Teacher Education.
Oral presentations in the form of seminars or verbal reports on projects are a key component in the teaching program for many courses at UTS. Assessment of these activities is not an easy task. An example of the criteria that may be used in the assessment of a presentation is given below. In some cases, staff use assessment from fellow students (peer assessment) in addition, or as a supplement, to their own assessment. An example of a Peer Assessment format appears on Page 43.
The major problem in the assessment of presentations is that generally students are given insufficient instruction or guidance on the processes of oral presentation, and often not given the criteria against which that presentation will be assessed. If students are to be assessed on aspects other than the content, they should be given the opportunity to learn about those aspects and practice them before being assessed on them.
Structure
Did the presentation have an effective introduction?
Were the major points illustrated, explained and summarised?
Was there an effective transition between the main points?
Did it have an effective conclusion?
Was the timing appropriate?
Content
Was the content accurate?
Did the level of the content suit the audience?
Style
Did the presentation hold your interest?
Were presentation media used effectively?
An example of a score sheet for assessing case study presentations is attached. Special forms of presentation include debate and court cases. These methods are used in the Schools of Mechanical Engineering and Computing Sciences respectively.
|
Date: Group Number: Group Members: |
|||||
Your score is based on your ability to effectively explain the mechanics of a particular technique, the context in which this might best be applied, and the strategy you use to argue for or against that. |
|||||
| Out of ten, we have awarded your group/individual presentation a mark of: | |||||
| This is because we believe that your presentation achieved: (please tick only one of the four boxes) | |||||
| - at a superior level of quality (8.5 - 10) | |||||
| - at a predominantly good level of quality (7 - 8) | |||||
| - at a satisfactory level of quality (5 - 6.5) | |||||
| - at an unsatisfactory level of quality (<5) | |||||
| Su | G | Sa | US | ||
| Clear objectives and agenda | No clear objectives | ||||
| Topic covered in depth | Superficial treatment of topic | ||||
| Logically developed argument | Very difficult to follow | ||||
| Accurate presentation of factors | Much questionable or inaccurate evidence | ||||
| Rigorous critique of key concepts | Lack of demonstration of key concepts | ||||
| Original and creative approach | Little evidence of originality | ||||
| Appropriate length | Too long/short | ||||
| Multi media used well | Presentation was read | ||||
| Effective use of pictures, figures and tables | Pictures, figures and tables add little to thrust of presentation |
||||
| Very entertaining and interesting | Uninspiring, uninteresting and boring (specify which) |
||||
Several other forms of assessment were mentioned in submissions by Schools to the Working Party on Assessment. Some of these have been included in this section.
Aspects of analyses are included in the section on Essays
These can be developed at a range of levels. It is important that the students know what is expected of them in these cases.
For details on criteria, see Attachment A
The following are the criteria for a critique of a journal article in Nursing
Assessment will be based on the following criteria
Family Studies, Nursing post graduate subject
As methods of developing an awareness of processes, these techniques serve a very useful purpose and reveal a wealth of student knowledge and skills not detected by other methods, especially those that concentrate on outcomes. However, they can be very time consuming to grade.
For guidelines for preparation, see Attachment B.
The assessment of participation, or contributions to discussions, or any contribution from students "from the floor", is both a difficult task for the teacher, and inhibiting for the student, and should be avoided. Other forms, such as group presentations or peer assessment are more appropriate, and still require the students to contribute.
See the sections on Assessing Laboratory Work and Assessing Work Experience
The defining characteristic of self assessment is the involvement of the students in identifying standards and/or criteria to apply to their work, and making judgements about the extent to which they have met these criteria and standards. This form of assessment is commonly a supplement to teacher assessment of students, but in some cases it may replace it. This and further information is available in Implementing Student Self Assessment, Green Guide No 5, Sydney: HERDSA, by David Boud (1986).
Simulations, whether written, oral or computer-based, describe a scenario and ask students to respond as the scenes change, either as a result of what the student does at each step, or in a pre-programmed manner. Assessment can be based on processes (problem solving skills) and/or on outcomes (the fate of the patient, the aeroplane, the experiment, etc.).
| CRITERIA FOR EVALUATION - Book Review Assignment - Sociology |
||||||
| Criteria | 1 | 2 | 3 | 4 | 5 | Total |
| (i) Reflects original thought on the part of the reviewer (synthesis) | ||||||
| (ii) Adequately informs the reader as to the essence of the book reviewed | ||||||
| (iii) Demonstrates awareness of the social significance of the author's ideas | ||||||
| (iv) Relates the author's views to those expressed in text or class | ||||||
| (v) Does all of the above with economy and precision of style | ||||||
| - shows mastery of mechanics (gramatical structure, punctuation, usage) | ||||||
| - selects vocabulary for preciseness of meaning and appropriateness for style | ||||||
| - demonstrates ability to manipulate complex as well as simple sentences | ||||||
| (vi) Gives evidence of organisation according to logical basis | ||||||
| - shows unity of purpose and/or logically proves the thesis | ||||||
| - uses transitional devices to clearly indicate relationships between ideas | ||||||
| Total: | ||||||
Name: Grade: Comments: |
||||||
| Gardner. F/P. & Abraham, G.W., "A Grading Procedure for Student Writing", Teaching Sociology, 6 (1), 31-35. Sage Publications, 1978. | ||||||
| NURSING STUDIES VI A GUIDELINES FOR WRITING A REFLECTIVE JOURNAL (Adapted from: Holly, M.T. (1984) "Keeping a Personal - Professional Journal" Geelong: Deakin University) |
|
2. What was my role in the event? 3. What feelings and senses surrounded the vent? 4. What did I do? 5. How and what did I feel about what I did? Why? 6. What was the setting? 7. What was the flow of events? 8. What were the important elements of the event? 9. What preceded the event and what followed it? 10. What should I be aware of if the event recurs? |
At UTS there are at least three kinds of practical or work experience which are specifically assessed:
i. Industrial training in science-based
courses
ii. Nursing skills
iii. Teaching practicum
Procedures adopted for assessing each of these activities are discussed in the following sections, and illustrated with examples from three sectors of UTS.
Several areas within UTS consider Industrial Training as an essential part of their degree course and assess this component accordingly. The Applied Physics Department in the Faculty of Science is one such area. The following notes are taken from their procedures.
There are five components to the Department of Applied Physics overall assessment of Industrial Training:
1. Employer's report
The on-site supervisor is asked to complete a double-sided "Employer's Assessment Form" (Appendix 1), grading the student's overall performance in four general categories (Technical Competence, Teamwork, Responsibility and Initiative) on a five point rating scheme. Each category is subdivided into four or five more specific indicators, each of which is similarly graded.
2. Student log book
In addition to any normal records that may be required by the employer, the student must keep a weekly record of work done. This is written in a booklet provided by the Department (see Appendix 2 for the format), and is to be sighted and signed with comments every four weeks by the on-site supervisor.
3. Student report
A written report of at least 1000 words is required. Students are encouraged to make this as neat, colourful and informative as possible, to serve both as a model for following students, and to use as promotional material for other employers curious to know the capabilities of our undergraduates.
4. Student talk
A 10 minute talk, followed by two minutes of questions, is delivered to staff of the Department, fellow and prospective co-op students.
5. Academic supervisor's report
Each student has a staff member assigned as Academic Supervisor. During each assessment period, the Academic Supervisor makes several contacts with the student and visits him/her at least once, preparing a short report. This report does not contribute directly to the assessed grade, but serves to flag any potential problems and allows the Industrial Training Co-ordinator to suggest corrective measures. It also helps to gauge the level of difficulty of a particular position, so that future candidates may be streamed according to their abilities and the perceived demands of the job.
Grading details
A final grade is determined according to the following contributions from each assessable component described above:
Employer's report 50%
Log book 10%
Student report 25%
Talk 15%
1. Employer's report
Each of the four general categories are first given a mark out of 25 on the basis of the overall ranking:
A=25 B=20 C=15 D=10 E=5
These first approximations are then modified according to the grades of the more specific indicators: for each indicator graded at the overall level, no change is made to the mark out of 25; one mark is added for each level above the overall grade, and one mark is deducted for each level below the overall grade. The total for the four categories is then halved to give a result out of 50.
2. Log book
A score out of 10 is allocated, depending on factors such as consistency and the completeness of entries, employer's comments and neatness.
3. Student report
This is examined by the Industrial Training Co-ordinator and is given a mark out of 15 for content and a mark out of 10 for presentation.
4. Student talk
All academic staff of the Department attend the talks. They are given a result form and write down a mark out of 15 for each student. The average score is taken as the assessed mark.
Clinical placements are designed to complement and reinforce the learning that takes place in the classroom. The competencies identified set the scope of practice expected of students at this level and can be subsumed under four broad areas, namely,
Safe Practice, Knowledge, Sensitivity, Effective Management.
Students will be assessed in all these areas during their clinical placement to encourage and facilitate professional development and growth.
Details of the criteria and standards used in these four areas for a second year student are appended (Attachment B).
Knowledge and performance of nursing skills is also assessed in part through an 'OSCA'. An OSCA is an 'objective structured clinical assessment' situation which involves a series of testing stations with live simulated patients or mannequins or video/audio tape or written material presented with instructions to follow.
The stations represent clinical situations which require students to demonstrate application of their nursing knowledge base related to complex areas of practice such as disruption to mobility, disruption to stimulation and disruption to aeration through tracheostomy. Each station carries a specified number of points which form a total to be converted to a value of 30% for the subject.
Selection, implementation and evaluation of nursing intervention is assessed. Cleaning up and recording data is expected.
Students perform under examination conditions and are in the nursing laboratory for approximately one hour. The OSCA is scheduled during class time. Rotation from station to station is timed and movement occurs at the sound of a bell with a one minute warning signal. To enable students to assume their clinical roles most effectively, students are required to be in full uniform including pens and watches for the session.
The assessment procedures for the Teaching Practicum are under review. The School of Teacher Education is currently consulting with other teacher education institutions in the State to develop an assessment process which will be the outcome of a partnership involving the co-operating teacher, the tertiary advisor, and the student. At present, the School has evaluation record booklets which consists of three sections.
(i) The first section gives information for the guidance of advisers.
(ii) The second section has a list of specific competencies for the particular practice together with an assessment scale.
(iii) The third section has a form relating to the evaluation of the student's teaching competence and professional development. Included in this form is a summary of assessment.
An example for the Physical Education program is included as Attachment C.
Adapted from information supplied by Associate
Professor Peter Logan, School of Physical Sciences, and by the
Faculty of Nursing.
Attachment A: Components of Industrial Training
Assessment
|
APPENDIX 1 Applied Physics Industrial Training
|
|
Use a 5-point rating for each aspect as follows: A EXCELLENT (has
shown an exceptional aptitude and level of performance) |
|
1. TECHNICAL COMPETENCE How would you rate the student's overall technical competence? Comments: |
|
2. TEAMWORK How highly would you rate the student's co-operativeness and overall teamwork? Comments: |
|
3. RESPONSIBILITY How highly would you rate the student's attitude of responsibility? Comments: |
|
4. INITIATIVE How highly would you rate the student's overall display of initiative? Comments: |
Appendix 2
|
||
| Week 1 ___/___/___ |
_____________________________________ __________________ ___________________ _____________________________________ ____ _________________________________ _______________________________ ______ |
Supervisor's Initials and Comments |
| Week 2 ___/___/___ |
_____________________________________ __________________ ___________________ _____________________________________ ____ _________________________________ _______________________________ ______ |
|
| Week 3 ___/___/___ |
_____________________________________ __________________ ___________________ _____________________________________ ____ _________________________________ _______________________________ ______ |
|
| Week 4 ___/___/___ |
_____________________________________ __________________ ___________________ _____________________________________ ____ _________________________________ _______________________________ ______ |
|
| SAFE PRACTICE | SENSITIVITY | KNOWLEDGE | MANAGEMENT STRATEGIES |
| CRITERIA - Understands and identifies the legal and ethical concepts relevant to setting - Demonstrates an awareness of the legal implicatioons of nursing actions - Acts to prevent or minimize risk to individuals and/or groups - Practices within her/his own abilities/level of learning - Demonstrates knowledge of policies and procedures that have implications for practice |
CRITERIA - Encourages patients' autonomy & recognises patients' rights - Uses strategies which affirm individuality of patients and their families - Communicates effectively with patients, families/significant others - Maintains a caring attitude - Respects the values, customs & spiritual beliefs of individuals/ groups |
CRITERIA - Identifies theoretical concepts underlying practice - Gives logical rationale for nursing actions - Performs appropriate nursing actions accurately & safely - Makes decisions that reflect knowledge of fact and sound judgement - Able to apply theoretical knowledge to practical problems |
CRITERIA - Demonstrates the ability to think critically & to solve problems in relation to patient care - Organise workload so that planned nursing care is effected - Collaborates with other members of the health care team in decision making & patient care - Evaluates goals & outcomes of care and adjusts care accordingly - Able to use resources effectively & efficiently |
| STANDARD Able to: - Use the clinical setting policy & procedure guidelines to enhance practice - Identify safety issues related to individuals in each clinical setting - Identify and apply if reasonable, interventions to enhance patient safety - Follow instructions: seeks help when necessary - Identify & discuss ethical & legal issues pertinent to each setting - Undertake designated specific procedures safely - Accurately record & report all nursing interventions |
STANDARD Able to: - Give examples of personal values & the possible effects of imposing them on others - Give examples of stereotypical assessments which may be detrimental to patient care - Explain what is meant by patient autonomy & the concept of patients' rights - Distinguish between judgements based on opinion & judgements based on evidence Communicates effectively by: - active listening - assertiveness if appropriate -congruent non-verbal behaviour - empathy - therapeutic touch |
STANDARD Able to: - Demonstrate knowledge of specific dysfunctional health patterns (outlined in subject outline) - Demonstrate & apply knowledge of nursing assessment & management related to above conditions - Identify ethical issues related to each clinical setting - Provide specific examples of how other subjects relate to clinical setting - State the requirements for the storage of S4 & S8 drugs - Discuss what is meant by confidentiality of information & the implications for patient care - Discuss role & functions of other health care team members |
STANDARD Able to: - Determine & justify priorities for care - Identify specific needs of patients and their families/significant others - Identify resources which may be used to enhance patient care - Demonstrate: co-ordinated behaviours - Achievement of major goals - Appropriate time management - Accurate reporting and recording |
| SECTION II I GENERAL SKILLS - PREPARATION Scale: N = Never; R = Rarely; S = Sometimes; O = Often; M = Mostly; A = Always |
||
| Specific Competency | Scale | Competency guidelines and comment |
| 1. Lesson objectives are clearly stated
in behavioural terms |
NRSOMA | |
| 2. Some knowledge (or estimate) of pupil's prior experience in relation to lesson content is indicated |
NRSOMA | Entering behaviour of pupils is considered.
Note: at this stage, this knowledge is not expected to be precise, ie. not the outcome of pre-testing |
| 3. Class management/organisation is clearly indicated in lesson notes. |
NRSOMA | (a) Spatial position of class is stated (eg. freely spaced; in circle; seated). (b) Size and number of groups given (eg. whole class, 4 groups of 5). (c) Planned use of working area is clear. (d) Working area is clearly delineated. States how equipment is to be distributed and organised. (e) Positioning of teacher is noted. (f) Details of how long various parts of the lesson will take could be noted. |
| 4. Necessary apparatus is listed | NRSOMA | Student teacher checks on availability and prepares necessary amount of apparatus (eg. ropes, balls, beanbags, tambourines, beatboards). |
| 5. Instructional procedures are clearly stated. |
NRSOMA | Refer to specimen lesson notes p10 in "procedure" column. |
| 6. Knowledge content is factually correct. | NRSOMA | Refer to specimen lesson notes p10 in "content" column. |
| 7. Selected learning material/aids and their uses are clearly detailed in lesson notes. |
NRSOMA | Aids may include chalkboard, flash cards, charts etc. |
| 8. Selected learning materials and lesson content are relevant in terms of lesson objectives. |
NRSOMA | Materials are appropriate to puils entering behaviour, contribute directly to lesson objectives. |
| 9. Evaluation procedures are clearly stated. |
NRSOMA | Evaluation procedures should check whether pupils have achieved lesson objectives. |
|
The following table issummary of the completed detailed competency ratings and assessment of professional development Use a tick where appropriate and comment to indicate strengths, weaknesses, remedia measures etc |
|||||||
| Unsatisfactory | Borderline | Satisfactory | |||||
| Competency | 1 | 2 | 3 | 4 | 5 | 6 | Please comment on each competency |
| GENERAL SKILLS Lesson Objectives |
|||||||
| Instructional procedures | |||||||
| Content knowledge | |||||||
| Content selection | |||||||
| Evaluation | |||||||
| TEACHING SKILLS GENERAL Effective introduction |
|||||||
| Class management | |||||||
| Communication | |||||||
| Lesson development | |||||||
| Chalkboard | |||||||
| Pupil evaluation | |||||||
| Lesson closure | |||||||
| TEACHING SKILLS SPECIFIC Narration |
|||||||
| Explanation | |||||||
| Practice | |||||||
| Questioning | |||||||
| Discovery learning | |||||||
| PROFESSIONAL ATTITUDE General attitude, co-operation Please comment |
|||||||
| Interest in school programme Please comment |
|||||||
| Involvement extra curricular activities Please comment |
|||||||
| Resource File | |||||||
Laboratory work can include one or various of the following approaches:
Controlled exercises, experiments, demonstrations
Laboratory kits and packages
Audiotutorial/laboratory method
Computer-assisted laboratory instruction
Case study approach
Project work
Participation in research
Each of the types of laboratory work can be designed to suit the following different purposes, although any one of the approaches to laboratory teaching may more readily fulfil one objective than another. It is good practice to deal with relatively few objectives in any one student exercise (major projects would be an obvious exception to this).
Knowledge and comprehension
Technical skills
Processes of scientific enquiry
(a) observing and measuring
(b) interpreting data
(c) identifying problems
(d) seeking ways to solve problems
Appreciation of the ways in which scientists work
Scientific attitudes and interests
Application of scientific knowledge and methods
There is a variety of ways in which student laboratory work can be assessed. Choice will depend on the objectives of the work. It is most helpful for students if a prime function of the assessment is to provide them with timely feedback. Regular indiscriminate assessment of laboratory work is a notorious burden for teaching staff as well as students. It can be replaced with fewer but more strategically planned checks and tests. Of the many kinds of assessment which could be used with imagination in the laboratory, some which are well suited to key objectives are outlined below.
Here, there can be direct testing of a student's ability to perform laboratory tasks under practical examination conditions. Purposes of laboratory work which require some form of practical test are technical skills and processes of scientific enquiry.
It is quite easy to construct reliable tests when the outcome of a student's work is a product which can be measured. Results can be compared with those obtained by staff experts or a standards laboratory. The use of instruments can be tested by asking the student to take readings or obtain printouts under some specified conditions. In more qualitative and descriptive areas, tests can also be successful with the key point being specification of the criteria for recognising competent performance.
Tests of technical skills and use of equipment can often be done quite quickly and lend themselves to use under examination conditions where students move from one station to the next at specific time intervals.
Aspects of scientific enquiry for which practical tests are essential are observing and measuring. However skills of interpretation and skills of problem identification and investigation can also be tested in a practical situation. These objectives can be tested under examination conditions by presenting students with phenomena to observe and or with experiments partly done by staff where students might take measurements, interpret results, make the next decision or carry out the next action.
Observations are essentially similar to practical tests but they are usually conducted as part of normal work not under examination conditions. Staff are asked to observe a student's performance in carrying out techniques or manipulations in the laboratory. The great advantage of observation is that it is, by definition, direct. Any alternative method when such objectives are important would be a surrogate, measuring by inference.
A reliable observation scheme can be constructed fairly easily so long as the criteria to be assessed are clearly agreed upon and a behavioural checklist can be developed. Assessing students' performance by impressionistic marking is not likely to be very helpful.
The first Attachment shows an example of a list of criteria developed for assessing manipulative skills. In each setting, it will be necessary to determine the standards to be met. Some skills may be considered essential and are best assessed with mastery standards, looking for 100% performance. These can easily be organised into a barrier examination whereby students would not be permitted to continue to the final examination until they have shown satisfactory performance on the mastery skills examination. (Opportunities for feedback and re-testing would have to be provided). Cases of skills performance in which it might be acceptable for a student to score lower than 90 - 100% would need to be carefully identified.
These are often used inappropriately in laboratory settings. If performance is required for a profession or for successful participation in a later course, a written test is no guarantee of standards.
However, useful written tests could concern knowledge of laboratory equipment. Tests could make use of photographs or diagrams of equipment. Students could be asked to read scales and determine measurements, to select correct settings, to choose suitable conditions for operation. They could examine an illustration and decide whether a piece of equipment has been correctly set up.
Other common uses for written tests include making calculations, error analysis, preparing graphs and other data manipulation techniques.
Attachment B shows examples of written tests on use of equipment.
In recent years, presentation of posters has become a regular feature of conferences in most areas of science and technology. These can be used for assessment of parts of students' laboratory work and, at the same time, introduce the idea of conference posters.
Design of experimental investigations is one of the most important parts of laboratory work. Posters would be a good way for individual students or groups to display their designs. Posters could also be useful for displaying the results of an investigative project. Attachment C shows a scheme for assessment of posters by staff or by other students in peer assessment.
These are probably the most conventional forms of assessment of students' laboratory work. Reporting mirrors an important activity in the careers of many UTS graduates and plays a useful role in building an appreciation of the ways in which people in those careers work and in the socialisation of students. The main cautions needed are to realise that both are indirect techniques and that both are post hoc reconstructions (the same is often said of scientific publications but in student cases, present a need for checking what actually transpired in the classes).
Other cautions include the need to protect against bias on grounds of gender and to be aware that it has sometimes been found that the factor most highly correlated with high grades is neatness, illustrating the need for clear criteria rather than impressionistic marking.
Attachment D shows a scheme which could be adapted for use in either written or verbal reports.
Dr Elizabeth Hazel, Centre for Learning and Teaching.
Boud D, Dunn J and Hegarty-Hazel E (1986) Teaching in Laboratories. Milton Keynes: Open University Press.
Dunn J (1986) Assessment of Students. In Boud D, Dunn J and Hegarty-Hazel E (1986).
Gibbs G and Jacques D (1987) Labs and Practicals. Oxford: Oxford Polytechnic Educational Methods Unit.
Hegarty-Hazel E (1990) The Student Laboratory and the Science Curriculum. London: Routledge.
A. Criteria for Observation of Manipulative
Skills (Englen and Kempa, 1974)
| COMPONENT | GENERALISED CRITERIA/PERFORMANCE FEATURES |
| Methodical working | Correct sequencing of tasks forming part of
overall operation. Effective and purposeful utilisation of equipment. Efficient use of working time. Ability to develop an acceptable working procedure on the basis of limited instruction. |
| Experimental technique | Correct handling of apparatus and chemicals. Safe execution of an experimental procedure. Taking of adequate precautions to ensure reliable observations and results. |
| Manual dexterity | Swift and confident manner of execution of practical
tasks. Successful completion of an operation or its constituent part-tasks. |
| Orderliness | Tidiness of the working area. Good utilisation of available bench space. Organisation in the placing of equipment used. |
|
In an experiment the 'dynamic method' is used to measure the
'force constant' of a spring. Your answer should include: (i) A table of data with error limits shown
A pencil and paper test for various experimental skills in data manipulation. next part involved the drawing in of the correct connections using he circuit diagram. This is quite a useful test item since it tests the ability of the student to convert an abstract concept into a practical reality. The circuit on the next page has been wired incorrectly. The circuit should have been wired according to the following diagram: Locate the wiring errors and indicate the location of each
error by drawing a circle around it. |
POSTER ASSESSMENT CRITERIA |
|||||||
| 4 | 3 | 2 | 1 | 0 | |||
| 1. | Self explanatory | Requires extensive additional information to understand |
|||||
| Comments: | |||||||
| 2. | Clear purpose or hypothesis | Unclear purpose (or no hypothesis) | |||||
| Comments: | |||||||
| 3. | Clear and appropriate methodology |
Unclear/unappropriate methodology | |||||
| Comments: | |||||||
| 4. | Clear and justified conclusion | Unclear/unjustified conclusion | |||||
| Comments: | |||||||
| 5. | Clear theoretical basis/clear theoretical implications |
Unclear theoretical complications | |||||
| Comments: | |||||||
| Total Score: | (out of 20) | ||||||
Other criteria could include: |
|||||||
| Visually effective and attractive | Visually ineffective and unattractive | ||||||
| Helpful/appropriate level of detail | Unhelpful/inappropriate level of detail |
||||||
| Means of data analysis and interpretation clear and appropriate |
Means of data analysis and interpretation unclear/ inappropriate |
||||||
| PRACTICAL WORK PRESENTATION
MARK SHEET Student/Team: Assessor Score for each criterion: 1. Inadequate 2. Barely adequate 3. Average 4. Good 5. Excellent |
||
| Criteria | Score | Comments |
| Account of the context of the study, its relation to other work, etc. | ||
| Statement of the precise purpose of the study | ||
| Account of the study and its methodology | ||
| Description of the findings, their analysis and significance | ||
| Statement of conclusions | ||
| References and follow-up reading suggested | ||
| Presentations skills, use of AV aids | ||
| Group skills, asking and answering questions | ||
| Total Score (out of 40) | ||
Project work is an attempt to introduce
genuine problems into education. In it students have to use their
initiative to identify problems they wish to solve or questions
they wish to explore, decide on the information, materials, equipment
which they need and how they can obtain them, use this information
to plan the work, attempt to solve the problem or answer the question
and present their results coherently. The way to assess project
work is indicated by the above list of activities which constitute
the work. Thus each of the following activities needs to be addressed
separately:
If only one assessment item (the project report or the project product) is used, it is important that students receive meaningful feedback during the progress of the project. Such feedback ought to assist the learning process and provide students with an approximate idea of their chances of passing the subject. If this is not done, students will concentrate their main efforts on the final product and not on the work preceding it. Most projects are assessed on more than the final report or product, by including forms such as oral examinations, observation, assessment of methodology, etc.
The great advantages of project work in higher education also create many problems. Projects provide a means through which a topic can be covered in depth (but the assessment becomes a time consuming and subjective task). The project requires students to undertake independent study and enquiry (but this creates differences in equality of tasks, and in the development common marking schemes). Projects encourage students to work together in groups (but student contributions are not always equivalent, and awarding grades is a problem). Some of these issues are addressed through the use of general assessment criteria. Examples are included at the end of this section. Group work is addressed below.
A proportion of project work is conducted by students in groups. Some of the advantages of group work include the enhanced learning resulting from combining the knowledge and experiences of several students, the fact that this type of work is more closely allied to the type of work many students will experience in their careers, and that the quantity of marking can be reduced. One of the major disadvantages felt by many staff (and students) is the injustice of awarding one mark to a group in which the contributions from individual students have not been equivalent.
A submission from the School of Teacher Education suggested that in group projects or assignments that have an overall assessment weighting of over 20%, students should be given the option of an individual mark.
One way of achieving this is to award the group one mark, but allow them to decide how that mark should be distributed among them. For example, if a group containing five members achieves a project mark of 60%, the group will be given 300 marks (60 x 5) and asked to allocate a mark for each person where no one can be given less than zero or more than 100 and the total is 300. Students will have to agree on the criteria being used to make this decision, and may learn some negotiating skills in the process. They can be asked to prepare for this negotiation by keeping minutes of meeting, diaries of events, drafts of contributions, etc.
Elton, L. (1987). Teaching in Higher Education. London: Kogan Page, p 100.
The first example is taken from the Department of Applied Physics Final Year Project. Before students undertake a project they are given an extensive document on various aspects of doing and writing up a project. The following extract is taken from this document and describes explicitly the way a major project is assessed.
Students will be assessed on their experimental, theoretical and laboratory skills and how they go about their project. At the start of a project there is a lot to learn and many students take 2 or 3 weeks to build up the confidence necessary to operate a piece of equipment or use a set of equations. In most instances students will not be assessed during this stage of their project. This is the time for learning from others such as the laboratory technician or student supervisor.
ÞStudents should be aware that there comes a time when they need to demonstrate that they can work and think independently. Students will be assessed on two components: the manner in which they carry out their project (assessed primarily by the supervisor) and the project report. The report is assessed by the supervisor and at least one other member of staff. When there are serious discrepancies between the two results, other members are asked to grade the report.
All completed projects are presented at a Project Assessment Meeting attended by all academic staff of the Department. All the marks awarded are discussed at this meeting and a final mark assigned to the project. In determining a mark, considerable emphasis is also placed on the degree of difficulty of the tasks on which the student actually worked and the quality of an achievement.
(a) Assessing the way the project is conducted
A number of factors are considered when assessing how well a project is carried out. It is worth emphasising that the list of criteria given below is not meant to be exhaustive. All projects are unique and any particular project may possess features not included in the list below. Alternatively, some of the items listed may be irrelevant. The assessment criteria therefore, must be tailored to the project.
Criteria include:
(b) Assessing the written report
Much of the assessment guide presented below was drawn up by Prosser & Cunningham (1983) to improve project report assessment at Griffith University. These criteria were used to develop those used at UTS. It should again be emphasized that there is no universal set of assessment guidelines; these have to be tailored according to the type of project undertaken.
The report as a whole
Criteria include:
Introduction
Criteria include:
Method
Criteria include:
Results
Criteria include:
Conclusion
Criteria include:
Prosser, M.T. & Cunningham, D. (1983). Making the Process and Criteria of Tertiary Science Projects More Explicit, Assessments and Evaluation in Higher Education, 8, 29-41).
Text: Dr Bob Cheary, School of Physical Science
Assessment in architecture is conducted in five stages. For these subjects which are creative and subjective, a number of viewings, sittings and opinions are needed to avoid prejudice in both personality and academic sense. This is best achieved through the following stages.
Stage 1. Tuition staff monitor students' progress - how quickly they develop ideas, how much they contribute to the group as a whole. The tutorial does provide for individual staff tuition, about 10 minutes each, but everyone listens/contributes/ criticises in a constructive way. It is not like going to the doctor. The art of tuition is to eliminate fatal flaws - see the essential strength of a scheme and build on that. Each student should leave UTS with direction and purpose, not a sense of bewilderment or despair.
Stage 2. Every project that is submitted is done via a 'viva' (See page 45). Students can speak for 5-10 minutes about their work and answer questions. This is no easy task for many. The viva has two important benefits. First it reveals levels of thinking that may not be apparent in drawings and models. The tutor is thus able to make a more accurate first appraisal.
Second it is the beginning of a lifetime of professional presentations that the architect is required to make. Projects needs to be explained to public interest groups, clients, government authorities and other architects. It is a vital part of the education of an architect.
Stage 3. After the conclusion of the 'vivas' all the tutors come together with the work and compare what they think is a pass/fail/credit/distinction etc. this way we argue the relative merits of work across the groups.
The outcome is a consensus mark that is issued to the students as a progressive assessment.
Stage 4. Before the progressive assessment is issued and the work returned to the students, a critique is organized with the whole year. Interesting examples are photographed and sequenced in two projectors and a dissolve unit. An audio-visual, either automatic or manual, is the method we have used for several years. It highlights the best of the best, reveals weaknesses in the program and brings all the issues up front for the whole year to debate.
Stage 5. At the end of the year a Design Review Panel is organized for each year. This consists of: Head of School, Professor, all the tuition staff of the particular year, 3 or 4 elected student representatives and other selected design staff. If consultants are important to the project then they may be present also. Student portfolios are discussed as a whole and the panel is able - again by consensus - to agree or disagree with the weighted recommended grade. The Design Review Panel determines the student assessment for the year as a whole. This mark cannot be altered unless the whole panel is reconvened and this is very rare indeed.
A set of criteria used to assess a particular project in Year 2 project in Architecture is attached. The completed sheet is returned to the student. The procedures used with this sheet are slightly different to those outlined above. For more details on the precise procedures, contact Steve Harfield or Winston Barnett, School of Architecture.
Text: Adrian Boddy, School of Architecture
Project Assessment Criteria: Steve Harfield,
School of Architecture
Criteria for Assessing Projects in Architecture
SCHOOL OF ARCHITECTUREUNIVERSITY OF TECHNOLOGY, SYDNEY
SUBMISSION REQUIREMENTS
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Problem-based learning is a term which is used to describe a number of different approaches to teaching and learning in which the prime focus is on a problem or problems rather than a discipline or body of knowledge.
The new perspective which is demanded by problem-based courses raises many issues concerning student assessment. Some of the features of assessment which have been emphasised in problem-based learning are:
(a) The importance of careful specification of learning objectives and criteria for assessment.
Often in discipline-based courses there is a premise that, because of shared assumptions on the part of staff about the nature of their discipline, it is not as important, as many educationalists would claim, to specify particular learning objectives and make explicit the criteria which will be used to make judgements about student performance. In problem-based courses there is a need for specification at all levels as the content is not indicated independently of what staff choose to include as problems. Problems are chosen with the multiple functions of:
(i) providing a stimulus for learning;
(ii) indicating important areas of competence in professional practice;
(iii) providing a vehicle for learning which is needed for appreciation of more sophisticated problems.
The selection of problems on this basis requires an understanding of both how students learn and what are key and common problems in practice. Staff are much less ready to make assumptions about these than they are to accept the conventional wisdom of their discipline. Thus they need to explicate objectives and criteria for themselves which in turn they can transmit to their students. Students can direct their learning more effectively when they know the criteria against which they will be judged.
The notion of assessment as essentially the task of ascertaining what has been learned at the end of a course is challenged by problem-based learning. Assessment is required to direct student learning and to give staff feedback on the curriculum as much as it is to measure outcomes. In problem-based learning there is a shift of emphasis from the sometimes exclusive focus on summative assessment, which the move to "continuous" assessment has produced in conventional courses, to a range of formative assessment practices concerned with making students aware of how they are performing and what they need to do to become more effective.
Assessment activities need to be varied and systematic so that skills can be developed over time and that appropriate methods are used depending on the stage of development of abilities. Assessment of processes and skills is needed, not just outcomes. The types and forms of assessment need to change along with the competencies being developed and the extent to which these become more complex and are integrated with each other.
(c) Assessment is for the benefit of student learning
Assessment is an activity which permeates learning and which needs to be undertaken for learning's sake, not just for the sake of accreditation of achievements. This has led to a much greater emphasis in problem-based learning on the role of self-assessment and the development of ways in which this can be facilitated. It can be argued that, educationally, the main role of any assessment activity is to encourage students to critically assess their own work. Learners themselves are finally the only people who can effect their own learning and unless other forms of assessment are directed to the end of influencing students to change their own learning practices they will have no educational benefit.
Process-oriented assessment methods include:
Tutor, peer and self-ratings (see pages 41 and 51)
Learning exercises, which are similar to, but shorter than, the five-stage processes described in the assessment of projects in Architecture (Page 75)
Outcome-oriented assessment methods include:
Written and computer-based simulations (see Page 51)
Essay exams (Page 25)
Multiple-choice exams (Page 19)
Short-answer tests (Page 32)
Boud, D. (1988). Assessment in Problem-based Learning, Assessment and Evaluation in Higher Education, 13, 87-91.
Swanson, D.B., Case, S.M. and van der Vleuten,
C.P.M. (1991). Strategies for Student Assessment . In D. Boud
and G. Felletti, The Challenge of Problem Based Learning, London:
Kogan Page.
Assessing
Mastery Learning
The aim of mastery learning is, as the title suggests, to develop in all students, a mastery of key components of a subject. Students are given clear guidance as to what must be mastered and are permitted as many attempts as they need, at the assessments used to test the understanding of that material, in order to demonstrate near complete understanding of it. The "pass mark" in mastery learning is usually 80-90%. Higher grades, if awarded, are determined using questions on the non-essential or non-mastery material, or by setting additional assessment for students who want the chance to achieve higher grades.
Advantages
Disadvantages
A variation on mastery learning which reduces
the assessment load for teaching staff is to encourage students
to achieve mastery, but to test their understanding or mastery,
and get feedback, using self or peer assessment. A range of practice
assessment items are still required, but the marking time is reduced.
When students feel they are ready,they have only one chance at
each formal assessment component of the subject, rather than unlimited
attempts, but they will still be expected to achieve a pass mark
well above 50%.
Any one of the forms of assessment outlined in the second section is, with the exception of some projects, rarely the only form of assessment used in a student's subject. In most subjects, the nature of the objectives (and in some cases, constraints such as students numbers) mean that an assessment package contains several of the forms listed in this booklet or referred to elsewhere.
One way of developing a complete assessment
package is to include a range of forms of assessment which best
match the range of mental abilities being tested. These could,
for example, be based on the range provided by Bloom et al. (1956),
as follows:
| Knowledge Comprehension Application Analysis Synthesis Evaluation |
Multiple-choise questions Short-answer questions Essay or problem questions |
This section contains some examples of complete assessment packages
used at UTS and the way they are designed to address the objectives
of the subjects. It also contains an example of one way in which
students are given freedom to select an assessment package which
gives them a greater chance to demonstrate what they know about
the subject.
It was noted in the earlier section on Assessment and Learning that assessment strongly influences the approach students take to their learning and in many cases defines the syllabus for a subject. Students will focus on those topics which are assessed at the expense of those which are not. It is therefore vital that what we are trying to achieve in our subjects (the subject objectives) are closely matched or linked to the assessments used. If this practice is clearly articulated to students, it can also help them in their learning. Some examples from subject descriptions from UTS courses which make the links explicit are given below.
The first example is taken from the Graduate
Diploma in Communication Management, from the School of Information
Studies
|
Objectives: 1. To analyse the distinctive features of a presentation as a means of persuasive communication. 2. To place the management of communication media in the context of relevant communication theories. 3. To analyse the communication characteristics of a range of media technologies. 4. To understand the different demands of radio, television and the press. 5. To understand the importance of "self" in the communication process. 6. To develop presentational competencies. 7. To develop competencies in the techniques of interviewing. Assessment components and their integration with objectives 1. Media File 30% Students will be required to produce one radio and television program and press feature column in terms of: demographics, content and style, media constraints, contact procedures and the personnel involved. This piece of work tests objectives 2, 3 and 4. 2. Interview 30% Students must conduct an interview for one of the following: the print media, radio or television. The complete interview must be accompanied by a report which details the planning, orientation and desired input of the interview on a specific audience. This piece of work tests objectives 2, 3, 4 and 7. 3. Presentation 40% Students will deliver a presentation of up to 15 minutes to a specified audience. The presentation is to be accompanied by a report detailing the rationale, aims, objectives, style of delivery and desired outcomes. This piece of work tests objectives 1, 2, 3, 5 and 6. |
|
Objectives 1. Broaden student's outlook and vision of economic phenomena by exposing them to recent developments in the different economic systems that comprise the modern international economy. 2. Develop an understanding of the impact on the Australian economy of the great upheaval that characterises the changing international economic system. This requires an identification of trading, investment and other business opportunities that these developments generate. 3. Develop an understanding of the principles of socialist economics and the workings of socialist economic systems in practice. This will focus on the important economies of China and the USSR. 4. Monitor changes in socialist economies and their domestic and international ramifications. 5. Monitor changes in non-socialist Third World economies and to assess the impact on the Australian and international economies. 6. Monitor changes in the major capitalist societies and to access the impact on the Australian and international economies. Assessment components and their integration with objectives 1. Class paper 20% The class paper is designed to test the theoretical and conceptual issues in the course in the context of specific case studies. 2. Essay 30% The essay requires an in-depth study of recent economic changes in the so-called socialist world and the implications for the Australian economy. 3. Examination 50% The final exam tests the major theoretical content of the course, the case studies of capitalists, socialists and their world economies and the implications for the Australian economy. Total 100% |
|
Specific Unit Objectives
Assignment 1: 20-30% Aim: to encourage you to develop skills in interviewing and to use the skills to determine how a teacher librarian provides for individual differences. Assignment 2: 30% Aim: To allow you to work with a small group of three to five persons in studying the research on providing for individual differences. Assignment 3: 30-40% Aim: To allow you to submit a case study, built on your first semester practice teaching, in an area which develops your skills in providing for individual differences as a teacher librarian. Overall Class participation: 10% |
|
Assignments (4 worth 5% each) will consist of laboratory exercises which involve the use of Macintosh computers. A typical format will be the construction and testing of assembler code programs, based on example code given. Students will tackle the assignments in pairs. As it is well known, students in this and other subjects exchange assignment material, and the grouping of students in to pairs formalises the exchange of such material and knowledge. The objectives of the assignments is to give students a view into the internals of a computer system by means of writing and debugging small assembler language programs. Also the debugger will be used as a means of making visible various internal components of the system. It is not possible, given time and other constraints, to produce competent assembler language programmers, however, on completion of the subject, many students should be able to read and understand, and write, simple 68000 assembler language programs. Understanding of the assignment material will be tested by small, informal quizzes, which will be scheduled from time to time. These quizzes will contribute 20% to the overall assessment. A final examination will be worth 60% of the overall assessment. |
Students are able to demonstrate that they understand their subjects in many ways. Some students are better at doing this in some ways than others. In the interests of giving students the best chance of demonstrating their knowledge, students can be given a choice in the assessment used to test this knowledge. This can range from Learning Contracts (see earlier section) in which the students and the lecturer reach an agreement on a set of appropriate assessment tasks, to cases where the lecturer offers students the choice of two or more assessment patterns.
An example of the latter case [From Mark Freeman and Michael Adams] is described in the following pages. This subject, Securities Market Regulation, in the School of Finance and Economics, has large enrolments.
In this subject (Securities Market Regulation)
students are assessed on individual performance, group performance
and group maintenance. The first component is based on a final
examination and mid-semester tests, the second on mid-semester
tests and "problem" assignments, and the third is based
on peer evaluation. The two assessment packages offered to students
(red and blue) reflect a different emphasis on the final examination
component, and students are asked to choose at the beginning of
the semester. The percentage weights are as follows:
| Package | Individual | Group Performance | Group Maintenance | Total | |
| Exam Tests | Tests | Problems | Peer Evaluations | ||
| Blue | 20 30 | 20 | 20 | 10 | 100% |
| Red | 50 10 | 10 | 20 | 10 | 100% |
There will be ten short, true-false multiple choice mini-tests designed to encourage students to learn key concepts and variables.
These will be conducted each week except for the first and last week. Only the seven best test results will be used for calculating each student's individual performance for tests. All tests will be closed book and cover cognitive information relating to the previous week's lecture and readings.
The second component of an individual's performance will be assessed by way of a final examination. This examination will be open book and relate to understanding the process, players and important factors and alternative solutions affecting a specific regulatory issue. Students will have to argue a position using the tools developed in the unit.
The same mini-tests given to individuals will be used for each group to discuss and then answer. This component is designed to encourage students to learn from each other.
In addition, each group will be given one problem for which they must present their solution. This component is designed to encourage students to list the variables relevant to the regulatory process, list and explain the alternative course of action and to argue a position using the tools developed in the course. Not only will this be evaluated by the lecturer, but each group not presenting will evaluate the group presentation. The marks for this presentation are worth 20% of the final results. Ten percent will be an average of these scores of the groups' evaluation and ten percent by lecturers. To aid your evaluation, the following rating guide is appropriate.
Criteria for failure
Reading the presentation
Inadequate coverage of major aspects
Obvious lack of demonstration of key concepts
Criteria for pass work
Adequate coverage of major aspects of assignment
Demonstration of core knowledge
Evident comprehension of key concepts
Clear and interesting presentation
Material well organised
Two media utilised in presentation
Criteria for credit work
The criteria for pass work
Evidence of relevant reading beyond core material
Critical comprehension and judgement of key concepts
Well-structured interesting and enjoyable presentation
Multi media utilised in presentation
Criteria for distinction work
The criteria for pass and credit work
Wider reading and gathering of appropriate evidence
Rigourous critique of existing theory or concepts
Evidence of originality and independence of thought
Extremely interesting and enjoyable, multi media presentation
Group maintenance (Peer evaluation)
Each individual will rate all of the other members of their groups during the final exam. This component is designed to encourage students to contribute to the learning of other students as well as themselves. Individual peer evaluation scores will be the average of the points they receive from the members of their group after the highest and lowest scores have been deleted. Assuming arbitrarily that: (1) the peer evaluation is worth 10 points, and (2) there are six members in a group, an example of the peer evaluation procedure would be as follows:
1. Each individual must assign a total of 50 points to the other five members in their group.
2. Raters must differentiate some in their ratings. (This means that each rater would have to give at least one score of 11 or higher - with a maximum of 15 - and at least one score of 9 or lower.)
Note: After the individual scores have been
computed, they will be standardised so that the average peer evaluation
score for all groups is identical. This will mean that the Peer
Evaluation scores will produce differences in grades only with
groups. As a result, you can't help everyone in your group get
a Distinction by giving them a high peer evaluation score. The
only way for everyone in a group to earn a distinction is by doing
an outstanding job on the individual and group exams and discussions.
Alternative assessment: an assessment component prepared especially for students who are unable to attend or complete the original assessment item because of misadventure.
Alternative examination: an examination arranged for students who are unable to attend or complete the final examination because of misadventure. Alternative examinations may be sat at the same time as supplementary examinations.
Assessment item: A component of an assessment pattern completed as a unit by students (e.g. a 2-hour examination). The item may contain more than one form of assessment (e.g. an examination with multiple choice questions and essay questions).
Assessment pattern: the collection of forms of assessment, their weightings and the timing of completion used in assessing the objectives of a subject.
Assignments: Components of assessment that are normally submitted from work conducted during the semester. Assignments can be formative or summative and of many forms (essays, short answer questions, computer programs, etc.).
Component of assessment: See assessment item.
Continuous assessment: An assessment pattern containing more than one assessment item with at least one item due for completion within the semester.
Contracts: Agreements between staff and students on issues associated with learning and assessment. For an example, see the section on Learning Contracts in Section 2 of the publication.
Criterion-referenced assessment: Where a student's level of performance is judged in terms of how well he/she achieves the objectives of the subject. The judgement does not depend at all on the performance of other students in the subject.
Feedback: Information returned to students on their progress in their course/subject. The information can be in the form of marks or grades, and/or qualitatively in the form of comments, model answers, suggestions for reading, etc.
Final examination: A test, quiz, essay paper, etc. set for students to complete in the official examination period at the end of each semester. Final examinations can be formal or informal.
Formal Examination: An assessment component conducted through the Academic Registrar.
Formative assessment: Designed to give students feedback on their progress towards the development of knowledge, understanding, skills and attitudes rather than assessment for marks or grades which are not given to students.
Forms of assessment: Ways of assessing students in which the method used matches the objective being tested. Examples of forms are given in section two of this publication and include multiple choice questions, essays, short answer questions, etc.
Grade: A symbol associated with the quality of a student's work. Recommended grades for use at UTS and descriptions of the qualitative relation between student work and the grade is presented in the UTS Assessment Procedures Manual. (see also interim result).
Informal Examination: An assessment component conducted by the lecturer/subject co-ordinator through the School, either during the semester or in the official examination period at the end of the semester.
Interim results: Student results determined by an Examination Review Committee which do not constitute a grade for the subject (e.g. result withheld, eligible for a supplementary examination) See Assessment Procedures Manual for more details.
Mark: A numerical value associated with the quality of a student's work.
Meaningful learning: Learning from which students develop personal meaning. Learning from which students have understood the information they are learning rather than memorising it specifically for the purpose of passing assessment tests.
Norm-referenced assessment: Where a student's level of performance is judged in terms of how well that student achieves relative to other students in the subject.
Objectives: What the subject/course is trying to achieve. The criteria against which a student's performance in that subject/course will be judged.
Reliability: The extent of the variation between grades or marks when any assessment item is graded or marked repeatedly. Assessments with low reliability provide marks or grades which depend largely on chance events.
Scaling marks: When combining marks from different assessment components, the spread of marks on each component should be the same. This is achieved by scaling the marks on the components. One method of scaling the marks is to give the lowest mark zero and the highest mark 100, then determine all other marks by a straight line conversion graph.
Subject outline: A description of the subject containing objectives for the subject, outline of the content, assessment details and how they relate to objectives, set texts and related readings.
Summative assessment: Assessment designed to be used to determine grades or marks.
Supplementary assessment: A second chance assessment component for students who just fail an original component.
Supplementary examinations: A second chance assessment component for students who just fail the final examination.
Validity:
Whether a test measures the ability being measured, or some different
related or unrelated ability.
Key
References
Andresen, L., Nightingale, P., Boud, D. and Magin, D. (1989). Strategies for Assessing Students (Teaching with Reduced Resources, No. 1.) Professional Development Centre, University of NSW.
Beard, R. (1970). Teaching and Learning in Higher Education. Harmondsworth: Penguin.
Billing, D.E. and Furniss, B.F. (Eds.) (1973). Aims Methods and Assessment in Advanced Science Education, Heydon & Son, London.
Boud, D. (1986). Implementing Student Self Assessment, Green Guide No 5, Sydney: HERDSA.
Boud, D. (1988). Assessment in Problem-based Learning, Assessment and Evaluation in Higher Education, 13, 87-91.
Boud D, Dunn J and Hegarty-Hazel E (1986). Teaching in Laboratories. Milton Keynes: Open University Press.
Boud, D. and Felletti, G. (1991). The Challenge of Problem Based Learning, London: Kogan Page.
Crooks, T. (1988). Assessing Student Performance, Green Guide No 8, Sydney: HERDSA.
Dunstan, M. (1971). A Guide to the Planning Writing and Review of Multiple Choice Tests, Tertiary Education Research Centre, University of New south Wales.
Ebel, R.(1965). Measuring Educational Achievement. Prentice-Hall, New Jersey.
Ebel, R.L. (1972). Essentials of Education Measurement. Englewood Cliffs, N.J.: Prentice-Hall.
Falchikov, N. (1986). Product comparisons and process benefits of peer group and self assessments. Assessment and Evaluation in Higher Education, 11, 146-166.
Goldsmith, M. (1991). (Ed.) Adult learning methods. Malabar: Krieger.
Gibbs, G., Habeshaw, S. and Habeshaw, T. (1986). 53 Interesting Ways to Assess Your Students. Technical and Educational Services: Bristol.
Gibbs G and Jacques D (1987). Labs and Practicals. Oxford: Oxford Polytechnic Educational Methods Unit.
Gronlund, N.E. (1982). Constructing Achievement Tests, 3rd. Edition, Prentice Hall.
Hiemstra, R and Sisco, B. (1990). Individualising instruction. San Francisco: Jossey-Bass.
Hills, J.R.(1976). Measurement and Evaluation in the Classroom. Westerville, Ohio: Merrill.
Hudson, B. (1973). Assessment Techniques - An Introduction, Methuen, London.
Knowles, M. (1986). Using learning contracts. San Francisco: Jossey-Bass.
Marton, F. Hounsell, D and Entwistle, N (1984). (Eds) The Experience of Learning. Edinburgh: Scottish Academic Press.
McKeachie, W.J. (1986). Teaching Tips: A guide for the beginning teacher. Lexington, Mass.: Heath.
Miller, C.M.L. and Partlett, M. (1974). Up to the Mark: A study of the examination game. Guildford, SRHE and Open University Press.
Newble, David and Cannon, Robert (1989). A Handbook for Teachers in University and Colleges. New York: Kogan Page.
Ramsden, P. (1988). (Ed) Improving Learning: New Perspectives. London: Kogan Page.
Rowntree, D. (1977). Assessing Students: How Shall We Know Them. Harper and Row, London.
Stratton, J.J. (1981). Recurrent Faults in Objective Test Items, Teaching at a Distance, 20, p.66-73.
White, E.M. (1985). Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating and Improving Student Performance. San Francisco: Jossey-Bass.