Continuous supervised course as examination
This is an example of a specific examination conducted at KTH. Principles for the approach that can be applied on other types of courses are described under the last heading of the page. In the specific examination approach, students solve a number of complicated tasks in an online training environment similar to a possible future workplace. The students' work is monitored continuously and assessment is based on a holistic perspective, where lower grades are obtained when the teacher helps the student to divide the problem into smaller pieces.
Course structure
The course takes place in a fictional company's IT infrastructure. During the course, the students must be able to carry out various attacks. The course usually has around 300 students and an intrusion detection system logs the students' work, sees all commands and packages that are sent. During the course, students are supposed to retrieve 18 "flags" from the company that correspond to various attacks that they learn by testing. A few students can do it themselves, but most need supervision. As support, they can then turn to a staffed help desk where they receive clues.
Assessment
The assessment is based on the fact that a completed task gives 10 points, and that each clue costs points. If the student wants help for 10 points, the entire solution is obtained. Challenges with the assessment are that the students can share clues with each other, or that the clues remain from the previous year. The IT infrastructure is rebuilt from year to year, but some older clues can still help the students.
Counteract cheating
The risk of cheating is addressed by the fictional company's detection system seeing if the students go straight to the solution (which is unlikely) instead of testing different ways to get to the solution. The students who behave as if they have received a clue in the wrong way are asked to conduct an oral exam where they explain how and what challenges they have solved. The detection system also draws teachers' attention to what students have not done, which can be a warning signal that is the basis for the student being called for an oral examination. The system has proven to work in such a way that relevant people are called to the supplementary oral exams and students usually subsequently request clues to really understand the elements and not be called to the oral examinations because they happened to use a method they did not really understand.
Among students who have had too little contact with teachers, either through the help desk or in the discussion forum, (often those who perform at a high level), a number are also randomly picked during the course to a complementary oral exam.
If a student still manages to cheat and deceive the detection system, the student should have enough knowledge to at least pass the course. The detection system reacts when a student completes a direct solution that would require someone completely different from the student to complete the course for the first time to cheat the system. The probability that the student has a replacement for the entire course is considered unlikely, but at the oral examinations the students must show identification.
Time required for examiners, teachers, assistants
Time in the course is spent mainly on preparatory work and giving individual feedback, via chat and during oral examiantions. The course begins with a start-up lecture but after that it takes place in the fictional company's IT infrastructure. During the course, a few lectures are given, mainly from guest lecturers, with a focus on inspiring and motivating students through real examples from working life.
Success factors that can be applied in other courses
The course has a specific context, but here are some factors that can be applied in other courses.
Turns the examination around and assumes that all students will receive the highest grade
Instead of studying how knowledge is put together from small details into a whole, where doubts can arise as to whether the teacher's help has connected the parts, the assessment is based on a holistic perspective where the teacher instead helps the student to divide the knowledge into smaller pieces.
Studies what the students haven't done in the learning process, but that they reasonably should have
By linking the assessment to the work and not the end product, it becomes clearer which students have really learned the course content. For example, it is highly unlikely that two students have the same draft papers, and it is reasonable to assume that the student first implements parts of a solution in the work of tackling a task.
Summons the students who have not actively participated in seminars, discussion forums, etc. to oral examinations
This way, all students will have had active contact with the teacher when the course is completed. Students who want to avoid an oral examination are encouraged to actively participate in the course.
Examines the students in the same way that they will use their knowledge in the future
The course and the examination are perceived as relevant, which can be motivating for the student. The examination also takes place for such a long time that it becomes difficult for the wrong person to have time to complete the examination in the student's place.