New IT system support for course evaluation and course analysis
To strengthen and facilitate course evaluation and course analysis, KTH has decided to develop a new IT system with a higher degree of automation. The new system will also provide better access to data to support developing the quality of courses. Here you can read about the project and see the timeline.
The project in brief
- The new system is Artologik Survey&Report, integrated into Canvas, which is adapted to KTH's needs. The current system LEQ will be phased out as the new system is introduced.
- The system will be tested during the spring/summer semester 2024 (VT24/ST24) and introduced live during the autumn semester 2024 (HT24).
- Course evaluations will be done automatically with clearer templates, which reduces the workload for teachers.
- Uniform templates are introduced for course evaluation, course analysis and the action plans drawn up for courses with identified shortcomings.
- Trainings and recommendations for the new system will be offered.
Background
The project to develop a new system for course evaluation and course analysis is part of a larger project to review KTH's entire quality assurance system for education.
Reviewing KTH's quality assurance system for education
During the years 2022-2023, the Dean of Faculty and the Faculty Council have been responsible for conducting a review of KTH's quality assurance system for education. The starting point for the review has been to evaluate whether the system is effective and appropriate, whether it is adapted to KTH's activities and organisation, and whether it creates value for KTH's teachers, students and management.
During the process of analysing the system, a problem statement has emerged. The problems relate partly to the fact that the system does not support the identification of shortcomings or does not require measures to be taken, and that functions in the line organisation are not involved in the quality assurance work. It also came to light that there are divided opinions about the benefits of the work with programme analyses, school reports and quality dialogues. The quality assurance work done on courses, for example through course evaluations and course analyses, has not been a clear part of the quality assurance system before either.
The starting point for the revised quality assurance system, including the work on course evaluation and course analysis, has been that it should support continuous work on following up the activities in order to identify strengths and weaknesses, to define areas of development and to follow up both measures taken and developments. Feedback to all parties involved, including students and doctoral students, is also an important part of the system. The quality system must also be resource efficient, both in terms of time spent by staff and from a financial perspective.
Quality assurance work for courses
At course level, the review showed that identified quality shortcomings can persist from year to year and that course analyses have not been perceived as meaningful by all teachers. There is also a lack of uniformity in course evaluation surveys and course analyses are rarely fed back to students. In addition, there is often a low response rate to course evaluations from students and doctoral students.
To improve the quality assurance work, the Faculty Council and Dean of Faculty have taken the initiative to prioritise course evaluation and course analysis within the revised quality assurance system.
Purpose
The purpose of the project is to develop the process support for course evaluations and course analysis with a system that simplifies the work for the teachers, ensures that all students and doctoral students are given the opportunity to answer the course surveys and that the data collected is made available to various stakeholders at KTH. Through the course evaluations the students and doctoral students will be given the opportunity, with a focus on course development and the students' goal achievement, to provide opinions on the design of the course, the examination and their own efforts. With the new system, course analyses will be easily published and available to read, which increases feedback to students and doctoral students.
The work also includes introducing uniform templates for course evaluation, course analysis and for the action plans to be drawn up for courses with identified shortcomings. By using the same templates, it will also be possible to compile and analyse data at different levels within KTH. It will also enable the relevant functions to get an overall picture of how the quality work with courses is conducted.
Goals
The project will result in KTH having implemented the new system for course evaluation and course analysis, Artologik Survey&Report, and developed new processes and support for the system.
Project goals
After the project is completed, seven project goals will have been achieved:
- New platform: Implemented a KTH-wide IT system for course evaluation and analysis.
- Canvas integration: Integrated the system with Canvas for more efficient access and management of course evaluations and analyses.
- Automated course analysis: Introduced automatic course analyses based on KTH's standard template with the possibility for teachers to add their own questions.
- Moderation of free text answers: Provided a function for moderation of free text answers to facilitate management for teachers.
- Publication of analyses: Implemented automatic publication of open course analysis (part 1) while internal analysis (part 2) is kept private for teachers.
- Data infrastructure: Created an infrastructure to utilise course evaluation and analysis data for quality monitoring.
- Data protection: Guaranteed that data storage and data management follows KTH's regulations.
Impact goals
After completion of the project, two overall impact goals will have been achieved:
- All KTH's students will after each completed course round have the opportunity to submit their views in a course evaluation according to KTH's course evaluation survey template, so that KTH can ensure compliance with the Higher Education Ordinance, Chapter 1, Section 14.
- All courses will have a published course analysis after the end of the course instance.
Process for course evaluation and analysis, step by step
Below you can see the intended process and order for course evaluation and analysis. The process will be tested and evaluated in different ways and is subject to change.
1. Creation of course survey
The course survey is created automatically for each course session, according to a standardised course survey template.
2. Opportunity to adjust course survey
Survey owners can adjust the course survey by adding questions and adjusting start and end dates.
3. Course survey published
The course survey is published in the course room in Canvas, for students who are registered for the course.
4. Course survey is answered
The course survey is answered by the students. A reminder goes out to the students who have not responded within a certain time.
5. Course survey is closed
After a certain time, the course survey is closed and can no longer be answered.
6. Review of free text responses
In order to filter out any threats or harassment, a review of free text answers that have been provided in the course survey is carried out. The review is included by default but can be deselected.
7. Reports are created
Two different reports based on the course survey are created:
- With free text answers. This report is only visible to the course coordinator.
- Without free text answers. This report is linked to the course analysis and is the report that will be published in the course room in Canvas and under About course.
8. Course analysis is carried out
The course coordinator makes a course analysis within 14 days of the survey being closed.
9. Reports with course analyses are published
Reports are published in the course room in Canvas and publicly under the course's About course page. The report is published automatically after 14 days, even if the course analysis is missing. If the course analysis is missing, the report only contains results from the course survey.
Time plan
| When | Activity |
|---|---|
| January-August 2024 | Development of the new system for course evaluation and course analysis, Artologik Survey&Report. |
| VT/ST 2024 | The new system is tested on pilot courses. |
| VT/ST 2024 | The system is evaluated and adjusted based on lessons learnt from the test on pilot courses. |
| From HT24 | The new system goes live and the old system, LEQ, is phased out. |
| During HT24 | Training for teachers, examiners and course coordinators. |
Information meetings
We will be hosting information meetings where you can hear about the project and ask questions. The information sessions are held over Zoom and are scheduled for the dates below. More information and training sessions will be held during HT24.
- 5 March at 10.30-11.30
- 19 March at 14.00-15.00
- 16 May at 10.30–11.30
- Around mid-June (more info to come)
Contact
Contact the project's communications officer William Källback Winter at wkw@kth.se if you have any questions about the project.
Frequently asked questions (FAQ)
Project group
Project organisation
The project is implemented by the quality coordinators at the University Management Office and the Portfolio Management Office for Sub-Portfolio Education (PMO/ITA), on behalf of the Faculty Council and the Dean of Faculty/Vice Dean of Faculty.