Data-driven assessment of qualifications to master's programmes
The project is a KTH-wide collaboration between departments to develop data-driven methods for the admission of external students to increase quality and reduce work effort. In October 2023, the project was merged with a corresponding project within the University Administration (VS) and will continue in 2024.
This development project started in 2023. It is carried out at the School of Engineering Sciences (SCI) within the framework of the change programme "Future Education at KTH".
In October 2023, the project was merged with the VS project Merit evaluation master's and collaboration with more schools began. The project now includes close to 30 master's programmes with programme directors (PAs) from SCI, EECS, and ITM, as well as the IT department and central and local administrative support at the schools concerned. The work will continue in 2024.
Background and problem definition
Before autumn 23, the Master's admission at the SCI School, a pilot project was carried out during HT22 with the overall goal of increasing the quality of the admission process while reducing the work effort for both PA and study administration. The pilot project achieved both goals while identifying several new ideas to continue toward the overall goal.
A) KTH has many international applicants for its Master's programmes. Evaluating these applicants currently takes considerable time both for KTH's administration and the programme coordinator (PA).
B) There is currently no validated method for how students' qualifications should be assessed to indicate their ability to cope with their studies.
Aim and objectives
This project aims to carry out the remaining work that the SCI School can carry out to drive the work forward. Collaboration with other schools and with the support services (VS) has the potential to lead to further improvements.
A) Less work for administration and PA. Followed up via time comparisons.
B) Better assessment for international admissions in 2024. Followed up via statistical analysis of developed models
Impact goals
A) Reduced time consumption for both administration and PA by using automated data management to quickly produce easy-to-handle decision support and a preliminary ranking of the student's qualifications.
B) Develop a more accurate and, at the same time, validated merit evaluation model.
Milestones
- 1 October 2023: New "Summary sheet" produced, tested and published for all participating programs.
- 15 January 2024: Validated admission model ready. Administrative support to support PA clearly,
- 25 January 2024: The Excel program for handling the "Summary sheet" has been tested and is ready. Continuously followed up by an internal working group.
Challenges
A) The assessment of the candidates' merits is carried out in three stages:
- Validation of language skills in English and grades from previous studies,
- Verification of qualifications in the subjects that are essential for the Master's programme applied for, and
- Evaluation of the applicant's merits to produce a ranking list.
All these steps require that the assessor (administrator or PA) searches for and opens relevant documents in the admission system NyA. With an average processing time of about 5 minutes per applicant, an administrator has time for about 100 applicants per working day. In addition, there is some unnecessary duplication of work.
B) A pilot study at the SCI School last academic year showed that performance data from previous years' students could be used to develop a more accurate and better-validated model for Master's admission.
Root causes
A) Applicants usually fill out a "Summary sheet" where all their qualifications are included when applying to the Master's programme. A pilot project conducted at the SCI School last year showed that it would be possible to automate the handling of the "Summary sheet" to a large extent to produce an easy-to-handle basis for decision-making.
B) Using data from a larger number of former students at KTH than in the pilot study is expected to provide a basis for an even better validation of the admission model.
Suggestions for improvement
The solution has been tested by 27 programmes (also from CBH) but has yet to be systematically evaluated. However, both Objective A and Objective B appear to be largely met. There are three different degrees of improvement suggestions:
- The connection between special entry requirements in the programme plan and display in the PA view must be thought through by participating programmes.
- A better understanding of how the PA works with the PA-view will reduce the PA's time spent per application even more.
- We didn't get the amount of data that would have been needed in time – currently too few data points for universities in some regions, leading to some justified concerns from PA. The process for producing data needs to be streamlined.
- The next big game-changer in the development is whether we can persuade UHR to redesign the NyA system for how to receive applications. This could reduce the administrative intermediate step between the Summary sheet and the PA and reduce VS work.
Results and lessons learned
- Efficiency measure: The number of minutes (average) to properly assess an application. Two PAs at SCI have reported that they are down to 3 minutes per application. The goal for the SCI school for next year is 2 minutes per application.
- Different stakeholders have different workloads at various times during the year. Coordination of major projects must take this into account, and someone must be responsible for this coordination (steering group under Future Education?).
- An essential part of the work is to figure out how the development work can be conducted efficiently. I have to dare to try it, though!
- The next big step in development requires collaboration with UHR (NyA) and other Swedish universities.
- Despite some initial frustration and a high workload, it has been fun and rewarding to be part of a project with the administration.
Next step
- The experiences need to be gathered and shared before next year's admissions. This needs to be done early so that the programme plans for the autumn semester 2025 have time to be changed.
- Who will be responsible for the follow-up in this project?
- It would be good if it is done by someone who has yet to be involved if we want to move forward towards UHR.