Setting up reliable online exams for large groups

During the Covid-19 crisis, the Board of Examiners of the Faculty of Arts discouraged teachers from offering online multiple-choice exams, because of the increased possibility of cheating. However, for large groups, essay or oral exams are not feasible. I therefore created an online exam with a mix of multiple-choice and short-answer questions. These questions were randomly selected so each student gets a different, but equivalent set of questions. I also took some additional measures.

Overview of settings

This is a set of settings within Nestor which I found useful to conduct online exams as an alternative to on-site multiple-choice. So far I’ve used this setup for the exam and resit of the course Statistics II in the academic year 2019/20 (106 enrolled students). These settings were decided upon together with the Board of Examiners of the Faculty of Arts and CLIQ. These settings were chosen as an alternative to proctoring tools, which were deemed too expensive and did not work convincingly in tests conducted at the Faculty.

Question type. A mix of multiple choice (in my case 29 questions, 58% of the grade) and short answer questions (6 questions, 42% of the grade). The questions can be created in a word processor, such as LibreOffice Writer or Microsoft Word, and then uploaded as a pool using Respondus. Two additional notes:

• Since students have access to studying materials during the exam, the questions need to be at a higher level in Bloom’s hierarchy of learning than in an on-site exam.

• In the analysis, the open questions came out, in general, as the most discriminative ones.\

Binary question sets. For each question there are two versions. Each student gets one of them randomly. Each question in the exam is a question set that points to two questions from the pool from which one is shown (the figure below shows an example). Randomise questions. The order of the questions is randomised.

No backtracking. Students cannot backtrack, i.e. after answering a question they cannot get back to it. This means they cannot change their answers should they be in contact with other students.

Time is limited. On the one hand time should be short enough so that students do not have time to finish early and help each other. On the other hand it should be long enough so that students have enough time and do not feel unnecessary stress.

Student pledge. Students have to sign a pledge before they start the exam.

Follow-up. A subset of students is called after the exam. During this follow-up conversation, the invigilator can verify whether the student submitted his/her own work.

In brief

Possible use cases. Online exams as an alternative to on-site multiple-choice exams, for courses that have too many students for other exam types (e.g. oral exam) to be viable.

Advantages. It is a viable online alternative to on-site multiple-choice exams. In the course in which I’ve used this, the distribution of the final grade was very similar to that of the previous year (when the exam was on-site):

Disadvantages. For teachers:

• Questions used for the exam cannot be reused since students can easily store (e.g. screenshot) and share them.
• This method cannot prevent one type of cheating: a student gives his/her Nestor login to someone else to take the exam.
• The information in the Test Analysis provided by Nestor is not correct due to the use of question sets. The chance correction needs thus to be calculated manually. In addition, for the calculation of chance it seems to make sense to take into account the number of unanswered questions (those that each student didn’t have time to answer) due to the strict time limitation, which means a different calculation for each student.
• Takes considerably longer to grade than an on-site multiple choice exam.

For students: Increased stress due to (i) not being able to rethink their answers (no backtracking) and (ii) the strict time limit. The time limitation might probably be detrimental for students that are slower in exams.

Costs. 0. Included in Nestor and UWP.

Learning curve. The first exam took a substantial amount of time and required communicating with exam support, Board of Examiners of the faculty and CLIQ. The second exam for which I tried this was rather straightforward to set up.

Additional information

Want to know more about digital examination via Nestor? The workflow document describes the different stages in digital testing.

Educational Support & Innovation (ESI) has built a separate tool for setting up randomized exams with essay questions.

You may also be interested in reading about preparing online written exams and how to use Respondus .