New IT system support for course evaluation and course analysis
To strengthen and facilitate course evaluation and course analysis, KTH has decided to develop a new IT system with a higher degree of automation. The new system will also provide better access to data to support developing the quality of courses. Here you can read about the project and see the timeline.
The project in brief
- The new system is Artologik Survey&Report, integrated into Canvas, which is adapted to KTH's needs. The current system LEQ will be phased out as the new system is introduced.
- The system will be tested early autumn semester 2024 (HT24) and introduced live later in HT24.
- Course evaluations will be done automatically with clearer templates, which reduces the workload for teachers.
- Uniform templates are introduced for course evaluation, course analysis and the action plans drawn up for courses with identified shortcomings.
- Trainings and recommendations for the new system will be offered.
Background
The project to develop a new system for course evaluation and course analysis is part of a larger project to review KTH's entire quality assurance system for education.
Reviewing KTH's quality assurance system for education
During the years 2022-2023, the Dean of Faculty and the Faculty Council have been responsible for conducting a review of KTH's quality assurance system for education. The starting point for the review has been to evaluate whether the system is effective and appropriate, whether it is adapted to KTH's activities and organisation, and whether it creates value for KTH's teachers, students and management.
During the process of analysing the system, a problem statement has emerged. The problems relate partly to the fact that the system does not support the identification of shortcomings or does not require measures to be taken, and that functions in the line organisation are not involved in the quality assurance work. It also came to light that there are divided opinions about the benefits of the work with programme analyses, school reports and quality dialogues. The quality assurance work done on courses, for example through course evaluations and course analyses, has not been a clear part of the quality assurance system before either.
The starting point for the revised quality assurance system, including the work on course evaluation and course analysis, has been that it should support continuous work on following up the activities in order to identify strengths and weaknesses, to define areas of development and to follow up both measures taken and developments. Feedback to all parties involved, including students and doctoral students, is also an important part of the system. The quality system must also be resource efficient, both in terms of time spent by staff and from a financial perspective.
Quality assurance work for courses
At course level, the review showed that identified quality shortcomings can persist from year to year and that course analyses have not been perceived as meaningful by all teachers. There is also a lack of uniformity in course evaluation surveys and course analyses are rarely fed back to students. In addition, there is often a low response rate to course evaluations from students and doctoral students.
To improve the quality assurance work, the Faculty Council and Dean of Faculty have taken the initiative to prioritise course evaluation and course analysis within the revised quality assurance system.
Purpose
The purpose of the project is to develop the process support for course evaluations and course analysis with a system that simplifies the work for the teachers, ensures that all students and doctoral students are given the opportunity to answer the course surveys and that the data collected is made available to various stakeholders at KTH. Through the course evaluations the students and doctoral students will be given the opportunity, with a focus on course development and the students' goal achievement, to provide opinions on the design of the course, the examination and their own efforts. With the new system, course analyses will be easily published and available to read, which increases feedback to students and doctoral students.
The work also includes introducing uniform templates for course evaluation, course analysis and for the action plans to be drawn up for courses with identified shortcomings. By using the same templates, it will also be possible to compile and analyse data at different levels within KTH. It will also enable the relevant functions to get an overall picture of how the quality work with courses is conducted.
Goals
The project will result in KTH having implemented the new system for course evaluation and course analysis, Artologik Survey&Report, and developed new processes and support for the system.
Project goals
After the project is completed, seven project goals will have been achieved:
- New platform: Implemented a KTH-wide IT system for course evaluation and analysis.
- Canvas integration: Integrated the system with Canvas for more efficient access and management of course evaluations and analyses.
- Automated course analysis: Introduced automatic course analyses based on KTH's standard template with the possibility for teachers to add their own questions.
- Moderation of free text answers: Provided a function for moderation of free text answers to facilitate management for teachers.
- Publication of analyses: Implemented automatic publication of open course analysis (part 1) while internal analysis (part 2) is kept private for teachers.
- Data infrastructure: Created an infrastructure to utilise course evaluation and analysis data for quality monitoring.
- Data protection: Guaranteed that data storage and data management follows KTH's regulations.
Impact goals
After completion of the project, two overall impact goals will have been achieved:
- All KTH's students will after each completed course round have the opportunity to submit their views in a course evaluation according to KTH's course evaluation survey template, so that KTH can ensure compliance with the Higher Education Ordinance, Chapter 1, Section 14.
- All courses will have a published course analysis after the end of the course instance.
Process for course evaluation and analysis, step by step
Below you can see the intended process and timeline for course evaluation and analysis. The process will be tested and evaluated in different ways and may be subject to change.
Process and timeline for course evaluation and analysis
Time plan
| When | Activity |
|---|---|
| January-October 2024 | Development of the new system for course evaluation and course analysis, Artologik Survey&Report. |
| Study period 1–2 HT24 | The new system is tested on pilot courses. |
| HT24–VT25 | The system is evaluated and adjusted based on lessons learnt from the test on pilot courses. |
| Study period 3, VT25 | The new system goes live and the old system, LEQ, is phased out. |
| During VT25 | Training for teachers, examiners and course coordinators. |
Information meetings
We will be hosting information meetings where you can hear about the project and ask questions. The information sessions are held over Zoom and are scheduled for the dates below. More information and training sessions will be held during HT24.
No more information meetings are planned at the moment, but training sessions on the new system will be offered eventually.
Contact
Contact the project's communications officer William Källback Winter at wkw@kth.se if you have any questions about the project.
Frequently asked questions (FAQ)
Project group
Project organisation
The project is implemented by the quality coordinators at the University Management Office and the Portfolio Management Office for Sub-Portfolio Education (PMO/ITA), on behalf of the Faculty Council and the Dean of Faculty/Vice Dean of Faculty.
Reference group
The Dean of Faculty has decided to appoint a reference group for the academic year 2024-2025 for the implementation of the part of KTH's revised quality system that concerns quality in courses. The group reports regularly to the First and Second Cycle Education Committee (GU) and involves the faculty board at each school when necessary. A representative from the group is also invited to the project group's meetings when necessary. See registration number V-2024-0667 for more information.