I write to share and invite discussion of something I presented at a conference at Nottingham Trent University last week.
I have been thinking a lot about assessment methods and their advantages and limitations for a chapter I am writing for my PhD thesis. For example, I could set a paper test and mark it by hand, as indeed I set one last week and will be marking it when I finish this post, and this allows me to give a personal touch and assess students’ written work but one downside is that I can’t return marks to students very quickly. I could return marks immediately if I used automated assessment, but then setting the assessment would be more difficult and I may be limited in the range of what I could assess. And so on.
I have been trying to classify these advantages and their paired limitations. My thinking is that by viewing different assessment methods as balanced sets of advantages and limitations we can justify different approaches in different circumstances and, particularly for my PhD, explore the advantage/limitation space for any untapped opportunities, which I won’t go into now (but ask me).
Here is my current list of potential advantages that assessment could access. These advantages are each something that I think that some assessment method can offer. My question is: what am I missing? I would be pleased to receive your thoughts on this in the comments.
- Immediate feedback. This is linked to learning from mistakes, confidence and motivation. It can also prioritise procedural learning over conceptual understanding.
- Detailed, personalised feedback. Though there is much disagreement in what I have read whether a human, who can respond to individual student work, or a computer, which will tirelessly generate worked examples using the context of the question asked, will in practice provide this.
- Individualised assessment. This is achieved through randomisation of questions and is linked to repeated practice, deterring plagiarism, allowing students to discuss the method of a piece of work without the risk of copying or collusion.
- Assessing across the whole syllabus. For example, computers can’t mark every topic.
- Testing application of technique. Whether students can apply some procedure.
- Assessing deep or conceptual learning. For example, open-ended or project work may require a detailed manual review to mark. This is linked to graduate skills development, etc.
- Easy to write new questions. Assume it is easy for a lecturer to write questions that students can answer (it isn’t, but we’re talking principle here). Difficulty is introduced by having to second guess an automated system, or having to second guess students to program misconceptions.
- Quick to set assessments. Assume that writing a test manually takes time. By quickly, I really mean choosing items from a question bank.
- Quick to mark assessments. Assume that marking by hand is not quick, perhaps unless the assessment is very short and student answer format very prescribed, in which case the assessment is limited. This is perhaps linked to problems of consistency and fairness when using multiple markers.
- Easy to monitor students. Clearly marking individual work from every student by hand will give great insight, but here I refer to the ability to gain a snapshot of how individuals and the cohort are doing as a whole with a concept, perhaps very soon after a lecture that introduced that concept has taken place.
- Perception of anonymity. I’ve read that some students are happier to make their mistakes if only a computer knows. This can reduce stress.
- Testing mathematical writing. Clearly requires hand-written work.
- Testing computer skills. Clearly requires use of a computer.
For example, then it might be possible to offer ‘Easy to write new questions’, ‘Assessing deep or conceptual learning’ and ‘Testing mathematical writing’ through a traditional paper-based, hand-marked assessment, but this would preclude, for example, ‘Immediate feedback’.
Similarly, a multiple-choice question bank might offer ‘Quick to set assessments’ and ‘Quick to mark assessments’ at the expense of ‘Assessing across the whole syllabus’ and ‘Assessing deep or conceptual learning’.
And so on. I have loads of these for different assessment types.
My question really is, is there anything missing from my list that might be delivered by an assessment method?
I was interested in the process Keith Devlin was talking about on some podcast or other recently, about getting students to mark each other’s work, anonymously, from a rubric. I think what you lose in immediacy, you gain in the students analysing “what mistakes has this other person made?”
You can also go back and give feedback afterwards as well.
“Students get to practice technique” isn’t quite covered by anything on the list, I don’t think. Often the only time students actually do the things they’re being taught is when they’re doing homework.
More on the perception of anonymity–it is a little less confidence breaking if you know that a computer has marked your answer wrong because it is wrong, as opposed to a human who introduces bias. If one constantly feels they are being judged on factors other than the actual question itself, this can be very deterring. I explicitly state this as such because your note suggests more of a shyness and an aversion to being seen by someone who can judge, where as fear of prejudice is quite another matter!
Thanks all.
@kensson: I think you mean calibrated peer review. I hope you heard it on this Math/Maths Podcast. I think you’re right that students can learn a lot from marking work.
@Christian: I have a section in my notes on ‘encouragement to practice’/’assessment-driven learning’ and had intended to include this under individualisation as “repeated practice”, but you’re right that it should be a separate item because students can be encouraged to practice by an assessment without randomisation. Thanks!
@Xittenn: Thanks. Yes, I have seen reference to automated marking being more objective, but hadn’t included it in my note here. But it’s in my thinking, indeed.
Exams can be used to verify that a student’s abilities match those demonstrated in coursework. For example, if someone submits good quality code as part of an assignment but is unable to tackle the most basic exam questions, there is good reason to suspect the student didn’t write the code themselves. It has been suggested that informing students in advance of this cross-check can reduce the frequency of such “out-sourcing”.
One thing I would like to add is that a well written assessment can act as a prompt to engage with the broader skills of scholarship (rather than merely rifling through the notes to find a similar example to transplant numbers into). It should encourage the student to revise the module as they attempt the problems rather than looking for the answers to isolated questions. This is often more readily achieved on paper-based work rather than computer delivered ones.
It’s also worth keeping in mind that some students struggle, usually for access/disability type reasons, to engage with computer delivered assessments (the same could be true in reverse too). For all students there is a small set-up cost in training them to use the computer assessment system.
Involving the students in the assessment process, either by a simulated peer marking exercise or getting them to mark a mocked up piece of work – and then explaining our marking of it – is invaluable as it can very effectively communicate our expectations. I also find this a useful exercise when I’m training postgraduate teaching assistants.
Also, I firmly agree that the communication of the mathematics should be an important component of the assessment; I have explicit marks flagged up for this on my assignments and, unscientifically, it appears to be working – both myself and my TAs are awarding more marks here over time.
exellent idea about assessment
This is all about thinking ability understanding some situation with understanding firmly agree that comunicationcan lead for a better solution ..