Integrity and Fairness in Assessment Through Computer-Based Examinations

This project aimed to create a framework for the development, delivery, deployment, and analysis of computer-based examinations. It involved a collaboration with instructors, students, and staff, to discuss the advantages and challenges of such examinations, and showcase a possible approach. This approach was successfully delivered, and its learnings disseminated.

The primary outcomes achieved were:

• The development of a free software tool to assist the creation of unique examinations for each student, thus directly addressing the issue of integrity.
• The dissemination of its results, through Teaching & Learning presentations, and two Academic Conference presentations.
• The creation of a set of guidelines for the creation, deployment, delivery and analysis of computer-based examinations in Universities.

SATLE priority area: Academic Integrity

Context and key activities

Although technology for digital examinations has been in use for decades, large universities such as UCD still rely on large hall, paper and pen examinations to ensure integrity, and offer limited support for digital examinations.

During the COVID pandemic, there was a substantial push towards experimental digital assessment approaches. However, with the onset of LLMs and AI assistants, those approaches were mostly not followed up, and many instructors reverted to pen and paper examinations as a means to ensure integrity, with most digital examination approaches having little to no guidelines or support offered.

This project helped address these challenges. Its main goals were: (i) the deployment of software to support instructors in the creation of digital examinations; (ii) the dissemination of its results; and (iii) the creation of a set of guidelines to highlight the challenges instructors face when choosing to use these examinations, and what is needed to support them.

Four Work Packages were considered.

Work Package 1 dealt with technical development and integration with Brightspace; its main outcomes were the delivery of freely available software platforms for Windows and Mac supporting the development of large-scale, computer-based examination questions.

Work Package 2 concentrated on the practical issues surrounding the creation, deployment, delivery and analysis of computer-based examinations; its main deliverables were large-scale demonstrations of the proposed framework, as well as a set of guidelines for such approaches.

Work Package 3 dealt with subject transferability; its main outcomes were several one-to-one sessions with School of Business colleagues, to investigate the applicability of the framework to various subjects.

Finally, Work Package 4 began investigating technological transferability of the proposed approach to other institutions and learning platforms; its main outcome was the identification of next steps required for a larger applicability of the developed software.

The project relied on several key stakeholders. It was primarily student-guided, with two students as direct collaborators. One provided the experience of sitting digital examinations, and the other the experience of supervising such examinations. A large-scale fully ethics approved study was also implemented, with the participation of 71 students, consisting of four trial exams. To investigate the validity and applicability of different setup variations, A-B testing was employed, where two balanced sets of students were provided with different setup; this provided valuable quantitative and qualitative data, with the students further providing extensive written feedback.

Four lecturers from three different Irish universities were also direct collaborators. They contributed their expertise with the use of digital examinations in their institutions, the challenges they face, and the outcomes they seek. A series of one-to-one interviews with six instructors at UCD was also conducted, showcasing the software developed, and collecting valuable feedback.

Three Research Assistants were employed. One worked on project management for the software tool developed; one ran interviews and demos with UCD staff; and one developed promotional media for the project (https://www.bsgenerator.org).

The main challenge faced was undoubtedly the lack of institutional agreed protocols and procedures for delivery of large computer-based examinations. While the project illustrated that the challenges of digital exams can be addressed, and showcased many of the advantages they provide, further work is required to guide instructors towards the adoption of computer-based examinations at scale and with an appropriate support framework.

Strategic alignment with national policies and relevant SATLE priority area

Although the work carried out in this project bridged the Digital Transformation and Academic Integrity areas, its outcomes addressed the latter very clearly.

One of the main obstacles to wider adoption of digital examinations is their propensity to fall easily prey to plagiarism and other forms of academic misconduct. This is clearly the case for online examinations, where students sit their exams from a remote location; while several proctoring approaches exist, such as webcam supervision and screen recording, they raise valid concerns about privacy invasion, and to some extent are easily circumvented. But even face-to-face, computer-based examinations are considered mostly unsafe, due to the unfamiliar challenge of supervising students sitting at their computers.

The software developed during the project addresses the use case of students sitting side by side in examination halls. Handwritten examinations require careful invigilation to avoid students copying from one another; however, with computer-based examinations, this is even more critical, as screens are typically positioned at a near-vertical angle and are light-emitting, as opposed to the horizontal layout of paper examinations.

The software addresses this by supporting the creation of unique examinations for each student. These can be carefully tested before the examination, and due to the digital nature of such examinations, any discrepancies between variations (such as substantial differences in difficulty) can also be carefully analysed after the examination has taken place.

The digital nature of examinations also allows for careful forensic analysis, detecting shared or prohibited IP addresses, connection and disconnection footprints, and similar indicators.

As such, the project aligns with the National Academic Integrity Network’s goal of developing national resources and tools for providers to address the challenges presented by academic misconduct. More specifically, it addresses two key points in its Academic Integrity Guidelines: point 3.2, Preventing Academic Misconduct and Protecting Academic Integrity, and point 3.3, Detecting Academic Misconduct.

Innovation in specified SATLE priority area

This project demonstrated innovation in Assessment Integrity through the development of software that addresses critical vulnerabilities in computer-based examination environments, while challenging the reliance on long established assessment integrity practices, such as invigilated ‘pen and paper’ examinations. It enables instructors to generate endless variations of exam questions, and also the means to thoroughly test those variations in advance of examinations. By creating tailored assessments, the software essentially eliminates the vulnerabilities associated with students sitting side-by-side in examination halls, while maintaining consistency in assessment standards and difficulty levels.

A key innovative feature addresses a critical institutional challenge: instructor control over assessment material through integrated Digital Asset Management (DAM) storage. This approach is deliberately platform-agnostic, protecting assessment content from the incompatibilities that arise during institutional system migrations. This was exemplified by UCD’s transition from Blackboard to Brightspace, during which significant content experienced compatibility issues. By storing assessment material in a format-independent manner, the software ensures long-term portability and institutional independence from proprietary platform constraints.

The project also recognises that assessment integrity extends beyond examination delivery to include feedback and learning outcomes. The software supports instructors in using their question variations and assessment data for subsequent feedback provision, enhancing the pedagogical value of assessments. Whereas this is typically hard to achieve for pen and paper examinations, particularly with large cohorts, the digital nature of computer-based assessment enables prompt provision of quality feedback.

The innovation reflects a shift from reactive, manual integrity management, rooted in traditional invigilation practices, to proactive, technology-enabled assessment design. The digital nature of the platform enables forensic analysis capabilities, including detection of suspicious patterns such as shared IP addresses and unusual connection footprints, providing evidence-based integrity verification.

The project moved beyond just showcasing the potential of such an approach, and effectively and successfully applied it to several cohorts in UCD. Over the course of the project, 2150 students sat exams designed using the software tool, from very large (590 students) to smaller (34 students) cohorts.

The project was also innovative in the way that it involved students in the generation/configuration of the quizzes. A set of 71 students volunteered to participate in a large-scale study, where different groups of students were exposed to different configurations, providing quantitative data to study the effect of different approaches, while providing extensive qualitative feedback.

Outcomes

Impact

This project has delivered clear benefits to both students and staff, demonstrating that computer-based examinations can enhance assessment integrity, fairness, and pedagogical value.

The framework and software were successfully deployed across both large cohorts (around 590 students) and small cohorts (around 34 students), using both institution-provided computers and students’ own laptops, in both open- and closed-book scenarios, and across several subjects. In all cases, grade distributions closely matched those of equivalent paper-based exams, confirming the validity of the assessment approach.

Feedback from students was extremely positive, highlighting the use of computers and a familiar virtual environment (Brightspace) as major advantages. Running exams in this manner supports practical demonstration of knowledge for students, helping to enhance their digital literacy skills. While the benefits of digital exams for students are well known, the developed software and framework help realise those advantages:

• Reduced exam day stress, with the knowledge that answers can be easily edited;
• Reduced anxiety about handwriting (readability and speed);
• Very strong benefits in terms of accessibility;
• Fewer transcription errors, ensuring fair and accurate evaluation of student answers;
• Potential for provision of quality, detailed feedback.

The software and approach researched also benefit Instructors. The tool allows instructors to store educational content in a platform-independent format, giving them full control of their assessment material. The ability to create variations truly opens up creativity in assessment design, and not only reduces some of the risks associated with students sitting side by side while taking examinations, but also opens up great potential for reuse of assessment components. It also helps instructors identify cohort-level failure points, evaluate question difficulty and discrimination, and predict grade distributions, supporting evidence-based teaching and the design of fairer assessments. Finally, by encouraging digital examinations, it facilitates advantages such as better organisation and storage of examination results, faster grading and reduced workload, consistent grading standards and easier matching of rubrics and marking schemes, better record-keeping and audit trails, etc.

The project’s approach has proven replicable and scalable. Kaplan Singapore has independently ran computer-based exams designed with the software, providing international validation of the framework’s transferability. Within Ireland, collaborators from other higher education institutions have expressed interest in applying the findings and tool developed, raising UCD’s profile as a leading institution in digital assessment integrity research.

Collaboration was central to the project’s success. Lecturers from multiple Irish universities, UCD staff, and students, all contributed to refining the approach, ensuring that it addressed the practical, pedagogical, and ethical dimensions of assessment integrity.

While institutional adoption of digital examinations at UCD and other higher education institutions moves at a slow pace, the project has raised awareness of the benefits of digital examinations through ongoing dissemination, Teaching & Learning presentations, and academic conferences:

• Invited speaker at UCD’s Technology Enhanced Learning for All (TELAll) seminar series 2023;
• Presentation at UCD School of Business Teaching & Learning Spring Workshop: “Assessment in the era of ChatGPT”, 2023
• Invited Panel Member at UCD School of Business Teaching & Learning Symposium on Artificial Intelligence and Re-thinking Assessment, 2023
• Nicolau M., O’Neill M., Higgins A., Munnelly J., McDermott, J. On the Advantages of Computer-Based Examinations, In: Academic and Research Integrity Conference, ARICI 2023.
• Higgins A., Nicolau, M., Flaherty, N., O’Neill, M., Munnelly, J., McDermott, J. Strategies for “AI aware” assessment design/re-design In: Academic and Research Integrity Conference, ARICI 2023.

The open-source software, accompanying guidelines, and documented findings are now available for wider use, ensuring that the project’s impact extends beyond its initial proof-of-concept scope and continues to inform the evolution of fair, secure, and data-informed assessment practices. Several UCD lecturers in the School of Business and School of Engineering have designed a variety of assessment components using the provided software, from weekly self-assessment quizzes to end-of-term exams, for several topics including statistics, mathematics, analytics, physics, and information management.

Resources

• https://bsgenerator.org
• UCD Teaching & Learning showcase