hi folks welcome to this eden webinar as we're waiting for people to come in i see about 40 people in so far please introduce yourself in the chat tell us where you're from and what the weather is like um in dublin today i can report it's terrible so i only want to hear nice weather if possible please uh so we're delighted to welcome you to this uh webinar about changing assessment due to covert 19 experiences and impact and we have some fantastic speakers lined up um who i'll introduce in just a moment i'm just going to allow a few more people to join salzburg croatia would you rather be there i can tell you so i'm just going to share my screen uh just a few things about the nap if you if you're new to to this web to this group or this webinar it's the uh eden network of academics and professionals and we do things like run networking events collaboration and also webinars like this and occasionally we also do tweet chats called eden chats so this is the steering committee just to briefly introduce us and also just to tell you a little bit about our speakers the first speaker is going to be dr monica ward then followed by professor steliano's hats of pangos from the university of london uh and presenting with uh dr linda and rayne cooper also from university of london and followed by dr inez gil aware arena from uned in madrid um so i'm going to hand over then to our first speaker which is dr monica ward so monica if you want to start getting your slides ready and i'll introduce reader bio so monica is an associate professor and assistant head for teaching excellence in the school of computing at dublin city university and a colleague of mine she has extensive experience in teaching and assessment in a range of subjects from technical to transversal skills she is a pioneer in the use of technology in education she advocates for co-creation and culturally responsive approaches with academics and students over to you monica grant everybody thanks for coming along today um it's uh terrible having to listen to your own bio but there you go and so oren has said who i am so don't don't believe the hype as we as we move through this uh presentation so uh first of all um just kind of give you a very brief overview of what i'm going to be talking about so i'm going to just look at academic integrity and kind of come up with a checklist for alternative assessment and then we're going to look at different types of attention assessment that we might have and some samples and tips so um i'll try and keep an eye on the chat as we go along as well i like to keep things as interactive as possible though it's time is limited so um the first thing about academic integrity it's what we all want in our assignments okay an assessment so um and when we kind of had to do the covert pivot last year it kind of really sharpened the mind of what we want in terms of academic integrity but i don't think we should kind of just park it there and say that was to do with that i think we should kind of bring these with us as we go along so obviously we want our exams to be valid we want to assess the students to make sure they know the knowledge but we have to be fair and to honor students so and we want to make sure that that those who do things honestly and correctly and don't cheat and kind of are are looked after okay um then we when in terms of an open book online non-vigilated exams there's a lot of concern so the academics are all worried about academic integrity will the students copy for another will they copy will they share their answers during the exam will they get the information from somewhere online they're all worried about that and we're all worried about the temptation for students okay so when they're in an exam hall and invigilated so vigilante means people walking up and down and checking that they're not cogging or copying from either it's a lot easier to be able to check that they're doing what they're meant to be doing okay so um so one of the things when we had to do the pivot we were moving from um you know a lot of the exams that we generally have and the easiest thing so to find you know rainy weather or define what an assessment is or to find bloom's sex animal it's really easy the students they're kind of memorizing their definitions and we have no idea whether they actually understand the words that they're regurgitating right so part of the the move was to move from remembering the lower order of bloom's taxonomy from the knowledge and remembering and comprehension understanding to apply it analyze it okay here's the framework for doing something now apply it and show us your understanding and it can be harder to set exams because you have to think of more challenging questions and if the students have only been doing kind of regurgitation or memorizing stuff to date it can be harder for them okay um but i would suggest that it's actually more authentic so in the real world so my um world is is programming um and sometimes our programming exams are okay you have access to nothing except you know a pen and paper and you have to write a computer program like there's no way in the real world that you'd be asked to do that if you're asked to write a program you will always have access online to kind of previous programs sample bits of code and you weave them in together to put to make things work so i think um moving to an open book exam is much more real world now i do accept there are some subjects where it might be slightly trickier to do but i i i do think it's kind of it is a step in the right direction um so some other benefits of the the digital uh space is the students can use digital tools and it's what they normally use in their kind of their general semester they're using things they're typing up documents in some sort of word process or they're using spreadsheets and all of a sudden in an exam situation we don't want them to be using word processes that seems mad to me right in the real world especially in my domain but i would argue in many domains you are using technology as as you go along and i don't know about your students but my students hardly ever write anything by hand sometimes even in a lecture they're taking you know they're typing on the computer they're not actually writing anything and the other major advantage in terms of marking is you can actually read what they write so sometimes it's kind of a guesstimate are they saying this um so i just wanted to look at some and we came up with a checklist for alternative online assessment so um we wanted to make sure that we were still checking the original learning outcomes um because we didn't want to go off and assessing something that wasn't required or we didn't want to miss any of the learning outcomes um we we said that the the new assessments had to have the same level of challenge as the original so we couldn't make them way harder because they're open book and they'd all cheat or we couldn't make them way easier because that wouldn't we couldn't stand over assessment that way and this was kind of a university-wide list not necessary for the school of computing students but we wanted to minimize the new computer and technical skills required in order to complete the work so are we minimizing it so um for students who wouldn't have used a particular type of software we certainly didn't want them to have to use as part of their exam okay um so we want to help to mitigate the risks um if the if the students couldn't submit to our online portal our online our virtual learning exam environment is called loop which is just an instance of moodle so what would we do if the students didn't have um access to online um internet issues okay um and then at a broad level we needed to consider it for different modules so there's different requirements if you have a technical module versus a non-technical module so if i'm taking testing the students programming skills versus their knowledge of the battle of x y z and they're out to be output of that that's very different um for some modules you know for modules that have working out bits for mathematical modules or programming modules you want the students to show their workings so even if they end up with the wrong answer that you can actually see that they had the right process and then some modules lend themselves to essay type answers again not too much in school computing but um obviously a lot of kind of things in the humanities domain would have those then we also needed to take into account the different stages of education so we could we have obviously different expectations of first-years versus master students and so we couldn't have called this is the one strategy that if it's all we had to be able to kind of cater for different learners okay so the general approach we adapted was we wanted to ask probing questions so the main thing was we wanted to see if they could apply the knowledge rather than could they remember it so um for example what are the five steps in design thinking so they would name you know whatever the the five steps are but now we assume that they have the five steps in a document in front of them and we want to see can they actually apply those five steps to a particular scenario okay and i would argue that's much more real worlds than getting them to remember the five steps because if they're in a job the managers are gonna say so without looking tell me what the five steps are rather okay here's our scenario go and apply the five steps and come back to me um so we're assuming that the students will know and have access to their notes so they will know in advance what's in the notes so if they've studied and they've been involved all along you know that that's fair i think and just to emphasize we're interested in how they apply their knowledge not that they have us okay so we don't want this kind of memory memorization kind of regurgitation thing so uh there's kind of two types of questions i'm just going to briefly touch on one type which is the you know in my scenario be software engineering type questions so this idea is that you'd set up a scenario and you would ask the student problems so what approach would you recommend and why so for example if dublin zoo want to um design a new it system uh what approach would you recommend and why so i'm assuming they can look through their notes and they say there's three different approaches to doing it and they could go with we'll go with a prototype approach and then develop from that or we'll go with an agile approach or we'll go with a different type of approach whatever applies in kind of software and then say how would they implement it so everybody would come up their answer to the second part would depend on the first part and their answer would be very different so it's not something that you can kind of take out of a book you have to be able to understand the different frameworks and apply them um it was so for my colleagues in computing um you know the habit would generally be to look at last year's questions change a little bit change the scenario a little bit but more or less the wording would stay the same so um i came up with some kind of general questions for them to as to feed and just give them ideas like so something like what is the most important of effective so they might know that there's three or four frameworks to do something and so like if you think take the battle of xyz so what was the most important strategy they adopted or which method was best when fighting in the desert or how would you design a whatever what changes would you make kind of given a particular scenario what other information that you need you're given this information why would you need to kind of further things along and a nice kind of get out of jail one is could you explain your reason to us so i went for framework a why did you do that i did it because it fitted the scenario or whatever um and then another thing that a lot of my colleagues found useful was smart quizzes okay so on uh on moodle and probably other vles as well you can have um you can have a bucket of questions so say if you have 100 students you don't want all 100 students to be getting the exact same questions so you would have a bucket so for example i could have a booklet about a question about population and i could have three questions there what is the population of albania china and kenya this is obviously a very bad example for an open book exam but the idea is that you would have three different questions and the student would only get one of them at random okay again for mathematical questions you can have questions of similar difficulty but each question a student when they see their exam paper they're getting a slightly different question and another important thing is to randomize the order of questions so say if i have 20 questions on my paper my question one is different from oren's question one okay so my question one might be auras question 17 but because born is under time pressure and i am as well to answer my question and my question one is china and hers is albania we kind of don't have time to kind of copy from with each other okay another important thing to do is um feedback for students is great but in a summative assessment obviously we don't want order to finish early and then if you say hey monica these are the answers to the questions so to make sure that you postpone feedback until all the questions are finished um so another colleagues wanted to have the kind of the option they have a multiple choice question and then you have to kind of flesh out the answer so um a question might be what is the standard deviation for avocado production in chile if the production values for the last x years are so you'd show the students a piece of data and they would say oh it's whatever and then you get the student kind of doing some sort of analysis piece so it's combining the multiple choice with um kind of a text piece which is exploring their learning um so you have to assume students have access to calculators and anything that's online okay um so the things that you can do is you can get them to fill out workings in a step-by-step approach i'll show an example in a minute you can force sequencing and so they can't answer question two until they've answered question one and they can only submit one so if aaron and i have both answered question one we can't go back and change our answers it depends on how strict you want to be and also you can enforce a time limit to make things trickier so this is an example don't worry that it's kind of computing related but what i have here i have an equation that you have to solve and the idea is i have all seven steps here but the students have to decide which order you actually do them in okay so it gets around the problem of students doing something in a different order i want them to clearly see this is step one this is step two this is step three okay i'm just showing some examples here this is another example of um so you know in things in computing are either one or zero so in this scenario here the student has to just fill into each box whether it's a one or a zero okay and each of these has a small amount of marks behind it and this is another one here so i want them to go through each of these different things don't worry about what i'm doing there but you can imagine some kind of mathematical calculation and for each of these things they have to fill in a piece of information so it allows me to limit what they do in terms of making it easier to mark and so just kind of look back again at academic integrity obviously it's a lot harder for non-invigilated exams so if somebody's not watching over the student it's harder to ensure that we are being kind of academically rigorous um there is plagiarism detection software on most um platforms so moodle has um plugins that we use there to to check for that and and i think this is kind of harder for introductory level courses so if you're asking students if you ask them something like two plus two equals four there's um not so many different ways that they can actually do that calculation and because they're introductory there's not a lot you can ask them and i think it's particularly challenging for programming and mathematical related courses so the kind of the the approach should be to um design your questions smartly making sure that you're checking for understanding rather than just kind of regurgitation of information and you want to check the student's analysis of concepts so you could have actually quite a big question and you know there might be certain differences between the answers and then your answer multiple choice answers can be nearly you know the same but kind of with subtle differences so just some tips to remember and ask questions that ask students to apply their knowledge not just check their knowledge you could relate it to project work and so say sayori did a project on kind of the botanic gardens and i did mine in the zoo so orna will only answer in botanic gardens and then i can't copy from her because her answers are different than mine you can use different variants of the same question um so some of my colleagues there's a tool on on moodle where you can import questions and you can actually randomly generate the elements of the questions with a little bit of coding and smart stuff too so instead of you know one plus two seven plus nine or whatever so you can do different things there some of my colleagues who are using data sets so arno would get the data set for flowers i'd get the one for trees somebody else would get the one for fruits um so there's no point in me copying i actually did detect some things unfortunately with students that barna gave the answers for flowers and she actually had the data for fruits didn't make sense at all and you know also always reserve the right to discuss with students after the exam and to check on their uh you know if anything suspicious happens i'm conscious of time laura so i'm just my last slide i think so um so i would say overall with positive outcomes so we encourage academics to kind of think outside the box in terms of how they assess students and some of them got them to do videos we had online meetings we can use interactive orals and just different ways of assessing the students instead of the normal kind of two-hour paper-based exam it also encouraged academics to reconsider their assessment practices and some of us won't go back to what we did before because what we have now is is so much better and but some are dying to go back to what they had before um and like in terms of the digital transformation in education you know sometimes we had to drag people's kicking and screaming to use our vle and now they're all on board and they're doing assessment stuff that they never dreamed they would use in a million years so i think that's it yeah and so i will uh stop sharing thank you very much monica very interesting uh i i think the the use of quizzes is very very interesting we have a few questions two questions for you before we move on to the next speaker and so the first there stays are i'd like to know your approach on the impersonating problem especially with second chance or extraordinary tests where some students have already passed and could be helping or even doing the exam with their colleague that has failed their first attempt so that's the first one okay i'll take that first so so in in our university all the students have to sit the test on whatever day it is um and if they don't make that and they have a chance to reset the exam um in august so that we don't have a scenario where orner does the test today and i have a sniffle and i said oh i can't possibly do it today i'll do it tomorrow we we don't have a facility to do that um we did put kind of extra things in place if a student had extenuating circumstances particularly in the you know in march to may last year when students had connectivity problems or whatever but in general we we don't have that scenario and so that's kind of interesting that you have that there cesar and one more which uh which are the reasons to force sequential quiz navigation presentational exams are able to switch as they like okay so i'm not too familiar with the term presenting examples examination either okay so and so in some in some subjects okay some colleagues did this either for questions or some of them had um a question and then you could you could only move on to the next question once you've finished question one or some of them did it for sections so i have all my mcqs questions first and then once you finish the mcq you move on to the more kind of text-based answers but you can't go back and change your mcqs okay so this is an integrity kind of reason yeah yeah okay i think uh that's very good thank you monica and we'll take more questions at the end but thank you very much uh and we're gonna move on now to our next speaker which is professor steliano's hats of pangos apologies for butchering your name uh stelianos uh works at the university of london centre for distance education where he's a fellow an executive co-lead for research and dissemination his expertise is in technology enhanced learning research informed innovation of academic practice and doctoral and postgraduate education management his research and scholarship includes learning design evaluation of online learning environments formative and technology enhanced assessment computer support and collaborative work digital literacy social media social networks in an educational context wow that's a lot of research areas and he's co-presenting with his colleague dr linda amraine cooper who i'll also introduce linda is head of the university of london's center for distance education she leads a team that supports the development of expertise in the field of distance education providing a focus for the development of high quality teaching and research in open and distance learning linda is also in the senior leadership team as director of strategic projects and leads at the pg learning and teaching he program at uol so welcome to both of you and i'll hand over to you thank you thank you linda is sharing his screen okay right hello everybody linda and i will share this slide presentation so uh from the title you must have understood that we are going to talk about this moving to online assessment uh uh evaluation at the university of london just to start with the overall context and i think this will give you a a good idea about what you will be talking about in late march 2020 the university communicated to all the students that the examinations in 2020 will have to move online because conventional examinations which would normally take at the sub centers around the world would not be possible due to their pandemic so 38 000 students from the university of london decent learners were registered to take over 100 000 exams in uh in the summer across 23 time zones so that's the kind of evaluation we are going to talk about linda can you please move to the next slide so very briefly very very briefly the structure of the talk uh we are going to talk about the purpose of the study some theoretical background then linda is going to talk about the methodology of the student voice i will i will touch on the examiner's feedback the pro program director's views and linda will talk about uh some messages some resources to support students to succeed in the next round of assessment 2021 and she will talk about the conclusions as well so the purpose of the evaluation was to provide a very detailed evaluation what happened last summer with this move to online assessment to identify the experience of the key stakeholders stakeholder groups and also to identify implications for the future of digital assessment at the university of london and finally very very important he had to do his lessons lessons learned to support the preparation for the summer 2021 assessment right so what i'm going to talk about here is some i will summarize some of the key outcomes of of the theoretical investigation we engaged with when we started this project um i'm glad that monica uh covered some of this stuff in relation to academic academic integrity for instance so um the research you looked at had to do with research was called pre-pandemic but interestingly they were very very fast because there was a frenetic base of people of academics moving their programs online but also a lot of debates and and discussions and depth declarations of what this means for uh for pedagogy so uh the recent debates on assessment in uh in higher education in the higher education sector revolve around changes and uncertainty of the future this investigated mainly whether 2020 marked the beginning of the end for impression fixed time paper-based assessment so quite a lot of base was around this area how uh how permanent would be these changes this transition to online learning or as monika mentioned some of these academics when the the unprecedented circumstances the pandemic were over whether they would move back to what they were doing for instance exam paper-based exams for their students recent research has also explored the relationship between the students performance and their preferences when they use online and offline assessments how to improve digital assessment practice a student motivation and engagement overall in previous research not exactly research that has taken place during this pandemic but previous research identified the students were comfortable with uh with online assessment they like online assessments they like the timeliness of the feedback they might be able to receive with online assessment the fight is quite convenient as well sometimes because of cost sometimes issues of accessibility the other kind of aspect that's very important there is that the shift to online assessment employing online vegetation and sometimes what we call proctored system has generated as monica mentioned quite a few debates on academic integrity so in this respect there are they seem to be to dominate dominant threats inside debates which are far from complementary what i mean by that is uh there is one threat that involves promoting creative design of authentic assessment the emphasis here is on authentic and clear guidelines to students about expectations around referencing and plagiarism in order to avoid plagiarism the other threat is more techie if i can use this expression provides technological a practical safeguards to protect academic integrity such as moderation of marketing text matching software and the use of other mechanisms for instance vivas to verify student academic work so there is an overall issue about equity uh of access also some academics talk about some researchers talk about the novel drop in assessment uh sorry an overall drop in achievement where students moved online assessment but there's no definitive kind of outcome or agreement in in public research about that and also there is a sweep as i just mentioned this overall kind of debate about assessment offenses and how you can manage academic integrity how can create awareness for plagiarism with the students in order to avoid these academic offences very serious issues the other kind of thing i need to mention is that we had in this exams where the students were asked to take these online exams were three exam formats and that was very important because it had to do with the window of submission uh there was a short submission period of up to five hours or immediate submission period which could be from 5 to 48 hours and finally there was a much wider submission window identifying resulting a long submission period of up to seven days uh over to you linda thanks diana so so just to go back to the so the university of london has 50 000 students who are studying online and distance with us and about 35 to 38 000 were due to take exams last may through june and so obviously we had to move those all online they would normally have taken pen and paper exams in exam sentences local to where they live and they're in located in about 190 countries so it was obviously going to be a really big change so the evaluation that we undertook then really considered these sort of four pillars uh that are on your screen here so student behavior so we were looking at how did student behave how did they behave did they do their exam which was one of the important questions about whether we were providing that opportunity how did they engage with the virtual learning environment where they were going to get their exams from we had some programs that were going to use a separate platform with a proctoring process on that um that was not successful in the sense that um i'll just detail that while we're here in the that students in different locations where they had less established bandwidths were not able to uh interact with the system effectively and it wasn't as as well developed as as we thought it was going to be so we pulled out a proctoring entirely and there was also questions about how do students submit the answers and and i know that there's already been some discussion of that so there was that student behavior section that we were looking at there was was student sentiment so we sent a survey to students out to that 35 38 000 student body um two weeks after their exam their set of exams finished so before their marks were confirmed we also undertook uh interviews and we had student research fellows working with us on the project as part of the interviewing team we considered student outcomes and that question of uh grades changes were there higher average marks or lower average marks we were very keen to know whether we disadvantaged our students badly could they not do the exam and did they do less well than they should have done uh in this process we look back over four years and considered distribution of awards as well as individual module marks or units of study marks and then we did a deep dive into the operational issues which surfaced some of the discussion that's already gone on about assessment offences integrity and the choices that were made by the programme teams so we were looking at it a hundred about a hundred thousand exams across about 120 different academic pathways and programs so a huge variety there computing english philosophy law undergraduate and postgraduates a really big variety of programs there we were also able to look at factors including location with students in as i said in 190 countries program gender age special exam arrangements um and exam type so just a brief bit about oops sorry about the uh responses we got about eight and a half thousand responses from the the the body that we sent out the survey to so this is the student body now responding uh a pretty even distribution of genders you'll see ages fairly well distributed in relation to the our demographic pattern which is that we have slightly older students than than on campus type provision study level undergraduate postgraduate mode of study teaching center independence so we have two models of student experience students can either study entirely online working through the vle or they can attend a local recognized teaching center that's part of our provision um so it was evenly split with students responding on those and then also we were able to get feedback from students who were taking those shorter exams which where the submission was between one hour and five hours and that varied by paper it was the same for everybody taking the same paper but it was different in different modules and then the medium exams which were up to 48 hours and then we had us just one program uh with a small cohort that did the seven day um return so i can't go into all of the details now and as you can imagine there was a lot there but i think it's what i've done here is trying to summarize um the impact on our students so the first thing is that 93 of exam events took place so that was 93 of the exams that we thought were going to happen that students were booked in for took place that's actually higher than our normal level it's usually around 89 so more students did more of their exams in terms of locations students in all locations engaged with exams that's not to say that every single student was able to engage with their exams of course there were some circumstance issues but overall there were no countries or locations where where nobody could get to their exam in the feedback 79 of them agreed that they were able to demonstrate their learning through the online assessment which we felt was a pretty good outcome given uh you know thinking back to what it was like in march we were setting up these exams and we're all working from you know our kitchen table etc as well as our students where as well some students were not fully equipped to engage with the format and the requirements of their exams so there are implications for training that will pick up a bit later on but some of the things there were if they had a 24 or a 48 hour exam what did that actually mean in terms of how much time they spent on that exam and some students found that was quite stressful so if they thought they had 48 hours they felt they needed to be working on the exam for 48 hours it also meant that things like word counts were quite important because if somebody's working on a response for 48 hours they can write an awful lot of words so we need to be careful about word counts um in communications you know there were opportunities for us to have enhanced communication i mean you know we we were really kind of not even swans because we were with a hundred thousand exams to deliver very quickly uh in a massive change like this there was some uh you know improvement that we could make um we did move the exams back a bit in the time so they didn't start exactly when they were due to start we moved them back a bit that did have some negative impact on students who might have jobs or family commitments who they had to make some rearrangements um assessment offences we had a higher number of assessment offences referred for consideration um and uh that then kind of bogged the system down a bit so it meant that we were later we'd move the exams we then had more uh exam offenses which meant the marks couldn't be released quite as quickly as normal so it did have sort of a knock-on effect into this academic year um so that's that's the sort of impact on students so overall they could do their exams they felt they were able to continue with their exams much of the open text comment was very uh positive about the experience how relieved they were to be able to do the exams how much they appreciated having the opportunity to do their exams given the circumstances that some of the stress was gone the stress of having to go into an exam center you know with the covert conditions but also the usual stress of having to go into exam centers but also some feedback to suggest that it was difficult to find a space broadband was an issue um you know those kind of things bearing in mind that all of our students were online students so they all had to have computers and some sort of connection in that's a sort of an entry requirement for us so it did reduce some of those digital inequalities for our student cohort i'm going to pass on to stelianos who's going to pick up on the results of the examiner survey and just i think silence you're just reporting on one or two findings there aren't you stelianos are you able to come back in you might be on mute yeah i was i was muted exactly okay excuse me uh very common uh right uh so um i will pick only on a couple of uh very important issues from the exam examiner survey that have to do with uh have a direct impact on the student uh aspect of the evaluation so what the the slide you are you're looking at uh was question four of the survey which asked them um to how this move to online assessment has affected the student performance and very interestingly a very positive the examiners have said that students have been able to achieve higher academic standards in the submitted work than the previous years which i think was very positive very very encouraging outcome for this kind of evaluation so the examiners were more positive about the transition rather than negative there were some other issues that were very important there as well um for instance students in the context of the paper-based exams that linda was talking about were submitting paper uh and handwritten exams whereas now all the exams were online so in typed exams the eligibility was much appreciated by the examiners the other kind of body of stakeholders group of stakeholders we're going i'm going to look at very briefly is the program director's views and then very important because the programming directors are there they belong to the program teams or lead the program tips that set the exams monica talked about quite a lot about this the format of the exam was very interesting for for me to hear so in terms of the problem director's views um there was a range of use on online assessment for uh for the future about the plan transformational assessment which has been accelerated by this uh pandemic a very important kind of outcome i think from the evaluation that for 2020 exams remained the predominant assessment format rather than moving to alternative formats of assessment for instance coursework which happened in uh in quite a few programs but overall exams remained their predominant assessment format the reasons that program directors use for that was that the the perceived academic rigor of the assessment process the recognition of the exams by professional bodies regional regulators and the impact and radical change might have on the coffin confidence in the quality of degrees in terms of pedagogy it was very important to mention that there was some rethinking of the end of module assessment pedagogy so in quite a few of the program directors that you talked with there was this adoption of alternative forms for instance coursework to complement or to replace the exam overall quite a lot of of good ideas there it is in the views about this shift from you the exams is the mainly it's mainly a measurement of learning to see the potential of these exams by transforming the content of exam as uh and discussing the potential of this for assessment for learning so the exam content for 2021 would move to open book exams and there was a in that dilemma i talked about in the the research there was a emphasis on redesigning an assessment rather than implementing huge stacky infrastructure that would monitor students over to you linda you're on me linda sorry obviously university of london haven't got the hang of this at all right so um definitely um some some real changes i think from from that emergency 2020 move online to now preparing for this summer's on online assessment so our you know about again about 35 000 students about 100 100 000 exams they started uh yesterday um about 7 000 students assessed in in the last couple of days now so we're moving forward with that and we we we spent a lot of time thinking about the findings from the evaluation and what it meant for planning for this summer and so these are the sort of things that we needed to uh to work at so communication and and you know having some plans around that faqs lots of webinars you know really clear sign posting and paperwork well-being student well-being that was really really significant and that's not just around the question of assessment it's around the question of uh living in a time of of pandemic i guess you know and we're all dealing with these issues of student well-being and supporting them but also motivation and peer-to-peer interactions and the way in which peers can work with with each other so that we've developed quite a lot in there um going back to that student behavior and i was talking about you know students not necessarily knowing what you do with a 48-hour exam for example so starting to to support students and our colleagues to explore how do you behave you know we're used to going into an exam center and known you've got to put your bag at the side and you can't have your phone and you can't have written anything up your arm all of that most people have had that experience growing up as a child going into unseen written exams but you know now we need to develop all the sort of behaviors that go around an exam um as well as obviously responding effectively to it enhancing technology making sure that our vles weren't going to crash that the systems were well signed posted those were important things changes to our general regulations really troubleshooting around the areas of academic integrity and assessment offences was very important and and very importantly as well is understanding uh the special those students who would normally have special exam arrangements um and how assistive technology uh can work there and we we did a quite a detailed study which i haven't got time to talk about really but looked at our large relatively large group of students who would normally have special exam arrangements and understanding what needed to be different for them so i'm just going to stop there ready to say um this is the sort of resource that we produce so students on the on the student portal have quizzes that they can to these not assessed quizzes these are quizzes and resources and materials um that help them to prepare for exams so that you can't see it really here but the study tips and and looking after yourself during the assessment period um there's a a a big piece on plagiarism and the rule for online timed assessments you know what we expect around citations given that we're talking about open book uh exams which i i think picks up with much of what's already been discussed today so in conclusion then we did it us and our students did their assessments and and they were able to demonstrate their learning and overall there was a a bit of an increase in in average academic outcomes for our students so they weren't disadvantaged unnecessarily globally there exam delivery parameters continue to vary to support different types of engagement and different types of student and different types of location and different types of subject discipline and within subjects the crisis has definitely accelerated the change and fostered a rethinking of assessment pedagogy you know to the point where we're all exhausted i suspect definitely spending time talking about designing for training understanding academic integrity issues are very important in this world of digital assessment and then to some extent going back to redesigning our modules and our qualifications going right back to that start of what are the learning outcomes that we're asking students to demonstrate through their constructively aligned assignments so it's um it's an exciting time it's um also you know like everywhere quite a challenging time with such large scale the references are in the slides and i presume everybody will get the slide so um we'll stop there thank you very much linda and silence the scale is staggering that really stands out to me um because like i felt a bit accurate as well i come from a similar background but i don't have that many students uh but but actually we had similar problems though so so very interesting and thank you for sharing your experience there's a few questions that we might actually leave till till the end if that's okay and i might introduce uh the next speaker which is inez uh so dr inez gil harena and apologies and as if i'm brutalizing your name my spanish pronunciation is also rubbish and so inez is an eden fellow and a member of the uh nap committee she's an associate professor at the faculty of education at uned in spain she coordinates the co-lab teaching innovation group battery at uned and her research in open distance education includes assessment and curriculum design over to you and is thank you orna and thank you all for joining us this afternoon um i'm i'm glad to see how and despite we are in different places um we are coming to the similar solutions and and concerns i will show you because i will present some uh well some some information from a study we have undertaken at uned um which is quite has some similarities with the one from the university of london center for distance education and also some of the guidelines my university has said are common to the ones that monica work said at the beginning well this studying i am going to present is not an institutional study but it's undertaken by by some faculty professors from my my research group and one of them is also attending the webinar daniel dominguez and our our concern at the beginning was to see what had happened in our course with each course together in the second semester and what had happened in our course when we had to change the assessment method last year very rapidly due to the lockdown and all the pandemic situation but then we we decided to extend the study to all the the university in the bachelor degrees um so what what happened last year and well first my university of ned is that is a distance education university and is the largest university in spain with a last year more than 125 000 students registered more of most of the of them part-time students and the demographics are similar to those that the previous percent has shown about their institution very diverse population different ages different background etc and the university had their first 28 bachelor degrees and this will be this is the focus of our study and the focus is also on the final examination which is the biggest part of the final score for us to our students there is also some continuous assessment and some assignments in the courses but the the final exam is very relevant for their the decision if they pass or not the course um commonly in before and after the pandemic the characteristic of the final exams is is that it's a it's course it's the faculty teams they decide the type of exam so they're they're very diverse type of exams and with different durations in less than two hours between one and two hours commonly and that all the examinations take place at the same time in the same course so they are synchronous in before in all the original centers face to face and now since june 2020 in the online examination platform and in this slide you can see the differences between the final examination until february 2020 and in june 2020 and the focus of our study is in in that semester in that first time that the university used the online assessment platform for the final examinations after that we have been using it a also in september last year in february we will use it in june this year and also in september so we didn't really know how long it will like the pandemic would last but we are in a similar situation now so we will be still using it and there are not face-to-face exams as they were before um currently so the characteristics before in the face-to-face exams where that they were taking place in different locations in spain and also in other places in europe and latin america mainly normally the examinations were without material and there were different uh control measures a well the students had to identify themselves before going into the classroom and they were immediately in big related exams they were faculty supervising all the the classroom and the examination process and also the students were distributed in the classroom so in a specific order so there were not two students from the same course sitting close to each other so they were distributed in the classroom and what happened in june 2020 well in in a very short period of time since march the university has had to set up a new system because it was not possible to have the face-to-face exam some some of the centers have very big a amount of students exam doing that some at the same time so it was not possible so the university said in our own platform is called avex which is our virtual the examiners examination classroom a virtual examination classroom um some in some courses now um in that call and also now materials were allowed so as they were more open book examinations but not in all cases so most most of the courses still had no material allowed during the exam and there were different control measures in the platform one was the identification well the students had to enter with their own password and also as some apps the other presenters said in their in their universities as well they examined the examine the different exams had different questions like random questions selected from a bucket of or from a pool of questions also there were there is a camera shots during the exam to the students to check if they are this is for identification and also the anti-plagiarism check and software to compare the students responses to the to the exams um in this and in this context our interest was to see if there was any impact on the student's performance when the examination changed to online and also what were the students perceptions about the process so about the academic performance indicators we have used the information from the data management office and we have processed it and this is a summary for all the degrees and you can see that all the rates were quite stable but they all went up in last last year in june 2020. and let's see them a bit more carefully the first rate and which is in my opinion the most important also because we are a distance education university where the assessment rate is not as high as in face-to-face universities um this this rate which is the relation between the students who were enrolled and those who who took the exam the students who were really assessed increase in all the degrees in the 28 degrees and this is very relevant because it it we can estimate that if the students complete the course and are assessed this would include their engagement and they will register the coming year which can solve a problem or a concern in distance education which is a profile rate and retention rates and in the specific degree that i coordinate the degree in social education you can see that also the assessment rate increased in last last year the second rate is the success rate which is the relation between the students who go to the final exam and those who pass the exam and it also increased in all the cases so this is a the the increase is not as big as in the assessment rate but it is also a positive the success rate increased in in that goal when the online assessment was just for the first time um and this is the case in the social education degree you see it is a higher rate but the increase is not as high as in the same rate and the the next rate is the achievement rate which is the relation between between the students enrolled originally and the students who passed the course and here also uh and consequently because this is related to the two previous rates this also increased um in in all the degrees you can see this is this represents the increase between the the rate the average rate in the previous years and the average in in june 2020 and this is the the achievement rate increasing in the degree of social education and finally the average mark which is the final score the students had in the in the courses and this also increased and in some degrees that the increase were very low and in the others was more than one point the the score is from 0 to 10 and the students have to have a minimum of 5 points to pass the course so the increase is in in some of the degrees is 0.4 14 which is very low increase but in others is more than one one zero five in the highest different okay and in social education the increase was not so high so what what does this say about the the students performance and to me the most significant is that more students took the final exam than in previous years and this was significantly higher with the relation with engagement that i mentioned before and consequently the achievement rate increased those who passed from those among those involved but the average mark so the final score increased but not so much and this is some of the points that can uh is leading to debate nowadays and it has to do with the academic integrity that we will talk about later and to complement this this study of the of the academic performance which is based on the data from all the courses we we and we took or we prepared the survey in in june 2020 uh originally it was going to be a survey only for our students in our course or in our degree but we made it we made it extensive to all the all the university and we got replaced from 714 students from 20 of the 28 bachelor degrees most of them are from the social science area i have to say and i have chosen some of the questions we made not not all the questions in the survey but one one question we had is a if the fact of the exams being online had influenced their decisions to take the exam or not and to a surprise most of them said it it had no influence so they were going to they took the exam uh in any case they were going to be assessed despite the type of exam for some of them maybe more of 10 percent it will it discourage them to take the sun because it was online and for some of them it encouraged them we will see later some reasons they they said for this another question we had is uh if they thought that the online exams were easier than the face-to-face ones and here minorities were did not agree they didn't think that online exam the online exams were easier we can contrast it this with the with the performance with the average score which we have seen it was higher but still they they didn't think that it was easier to have the examination online and finally we have for their preference and a minority but not a so big minority a bit more than half percent they prefer the online exams and still a forty percent of the students that replied to the the survey preferred face-to-face exams and in the open ended question of the survey um [Music] they they expressed some of the problems they had experienced it and the main mentioned problem was the time the duration of the exam then they thought it was not enough and for different reasons in some some faculty they decided to reduce a lot the duration of the exam and they said it was not enough or they had some issues with the writing in the computer if they had to write some open-ended questions and so the time was was a big concern and also the anxiety uh but mainly beforehand because it was the first time they were experiencing this this online system and they had a lot of worries before if the connection would fail if they had problems with the software or with any technical aspect so it was more anxiety before doing the exam because afterwards the main opinions were that it went well the platform was easy to use so regarding the preferences um for those who prefer the online exam a very big reason was the mobility like they didn't have to to travel to the regional center uh if they had mobility issues and mobility problems it was easier data exams from wherever they were family conciliation so these were some reasons they they expressed for preparing the online exams and for preferring the face-to-face face exams the thing is that the replays they gave to the survey they were not really related to preferring face-to-face exams for themselves but because they had had problems with the online exam in that same in that first call in june 2020 like bad experience because they thought they think they thought the the time was not enough they had problems with the internet or they didn't have a proper place to to take the exam quietly and they prefer the exam classroom in the regional center um and also a minority of the students were claiming for a more diverse assessment type not only the final exam um like having more assignments during the semester etc so having this information uh what are currently the faculty concerns and this is very common to what the previous presenters have said the main one and is seen in the debate every day is about the academic integrity or plagiarism we don't have really information about the number of cases of plagiarism that has been detected and effectively um [Music] not just a suspicion but confirm plagiarism but those cases that have happened uh have been have made a lot of noise like um especially people sharing exams or having a like wassup group sharing the examination making the exams in group together not individually so this is the main concern and the other one is the workload first in in designing exams that are more appropriate to avoid plagiarism and um and to have a more authentic assessment and also the workload in marking because as the assessment rate increased all the faculty will have more exam exams to mark at the end of the semester so it's the workload was really high last last year in the during the pandemic and the university has sent to all the faculty some guidelines for how to design the exams and these are common to what the previous presidents have said they are recommending the open textbooks open book exams to increase the the bucket of questions for have for run have random randomization to avoid this very basic in the bloom taxonomy questions for just remember etc uh literal and to adjust the duration of the exam so this the students can focus on their exam and they don't have time to share with others or to do some misbehaviors um so now the question is what will happen and and some faculty i don't know we are seeing this as an opportunity to redesign all the overall assessment process and to try to rethink the role of the final examination which we still have need to have this is a requirement from the ministry we would need to do a lot of uh things to to to help to remove some final exam um and also we have to be to be remind the number of students we have for instance in my course i have 900 students so it's very difficult to to have a very um formative and continuous assessment with 900 students but we still have the opportunity to rethink the overall assessment process and to think on using diverse assessment methods and the big question uh is if we will go back to the face-to-face exams a as soon as the pandemic allows it or it will be combining the online exams and on the face-to-face or what will happen and we don't have the answer to that yet um well thank you very much for for your attention thanks it is very interesting stuff i see i see you had the cove events which is what we were calling at last june that all the marks seemed a little higher um but what's one thing that is very interesting about all three presentations was the similarities and and experience and approach and really really quite similar so we might just we're we're a bit over time but we'll just keep going and we might just take a few week questions if that's all right there's a few there in the chat and i see monica's jumping in there answering one about students with disabilities i think stelianis and linda you had some particular approaches around dealing with students with disabilities do you want to briefly answer on that sure oh yes i mean absolutely so students who would you know we have a variety of different arrangements that are made for for students depending on their particular needs so uh extended time so you know we just applied that as well so they would have more time if they if they needed it but students for example who might have previously had somebody who did the writing for them ascribe that was much more challenging not not in terms of our system but because of covid because they could probably couldn't have a scribe with them so one of the things that we're looking into in quite a lot of detail at the moment is the use of speech to text software to support students there and alternatively text to speech software and on one of the things we've identified is that not all students know how to use what are quite common tools on on most of our computers now so actually for all students we we're we've produced a little teaching aid to help them understand how they can use that um the other thing was uh you know where students might have had um somebody who read the exam papers you know uh to them and so on that we were able to facilitate because we could use zoom or phone calls and so on to to help that happen but um and then a number of students who would normally have had uh difficulty you know physical difficulty maybe getting into an exam room or traveling to the exam center and so on actually their feedback was that this was really liberating and they never wanted to go back to exam centers and and pen and paper um but definitely you know a really wide variety of different needs um and some of them were had better experiences with the online assessment and some had worse experiences but of course underpinning all of this was the fact that students were themselves getting covered or their family members were getting covered so we were dealing all the time with you know and of course the staff were as well particularly as our staff with our teaching centers are around the world at different points different people were getting sick so um it's you know it's been kind of an ongoing rollercoaster of discovery i think is one way of describing it thanks linda and i see monica was typing away very actually similar how how students were handling it in dcu and actually that was a great uh point there monica about students using their own laptops which are set up with their different assistive technologies and because i think before did we used to make them use borrowed ones or something yeah well no i mean you pre-code they'd have to go into a lab where they'd have the software installed but it was so clunky like they were all sitting in a desktop and they'd have to cop they'd have to copy it onto a um i can't even remember it was cd-rom or something that's something ridiculous like i mean especially for computing students like that's neander and then i think we might have moved to um usb and i don't know more recently we might have been getting kind of emailed or put up somewhere in the system or whatever but and so i because i i'm conscious like of the disability students like we have a colleague who's completely blind so he'd be rubino for one to the front of our minds and that um i don't know if students are hard of hearing um i i think for some students with disabilities the fact that they could be on their own in their own environment using the their own setup of however they they learned i think that was actually quite liberating for them we would have quite a few students on on the autistic spectrum um you know especially in computing um so some of them would have sort of you know anxiety and stuff like that so they'd have to be in a room with their own and um i think the fact that they were at home or whatever it was was great i mean having said that i mean i know students who were really worried like they'd only one laptop they had another sister doing an exam their parents needed it for work some of them had to mind their older sister looking after granny and so it wasn't all sweetness on light definitely that was the same for our students we asked them those questions in the evaluation and we got a lot of open tech text responses on that um you know the challenge of of having a room of your own to in order to do the exam and not having other people kind of coming in not having noise and all of that was was a challenge and broadband was a significant challenge um students for example some of our students in bangladesh had real problems with with having sufficient broadband and then um with having many of them having to use their phone to tether um and then you know the costs of that etc so so there were many challenges yeah we with students in india as well and they were saying our we've lost internet in our region for the next four hours and what are we going to do that timing of the exam was also challenging yours was larger scale than us but we would have students in kind of vancouver over to india and whatever and just finding a time so some students are saying well i had to get up at two o'clock in the morning to do the exam and you know you can't you can't please everybody that's one of the reasons why we use the 24-hour 48 hour window with people having for some of the exams once you clicked on the exam you then had three hours or two hours or whatever you could click on it any time in that 24-hour window um it was your it was up to you to work for a time that worked for you so that was okay but that then does present the opportunity for students to screencast what's on their exam paper and send it down the road to somebody else so you know all of those uh issues kind of relate back to these integrity questions but the work that you did monica around designing exams with application and really using higher order skills are exactly what's so important i think as we move forward and you also actually said something very important about the exam so just because it's a 24 exam doesn't mean they have to work 24 hours on it that you're expecting you know a report or a case study or um whatever so i gave my students um an eight hour exam um and you know so they knew what was going to be on it i was going to ask them about design thinking and i was going to ask about some business model canvas or whatever but they didn't know the context it was going to be applied to and you know if if you copied the same thing as orna and you suddenly it was about to open zoo but you all started talking about rhinoceroses instead of picking birds or you know african animals or whatever then you know you'd see the similarities yeah kind of emerging yeah yeah yeah but uh what comes across i think from all three talks it corrects me if i'm wrong that uh uh the emphasis that we as educators support in redesign and assessment uh so okay all the technical infrastructure about plagiarism and proctoring and online visualization software and all this kind of thing might be important but without the the important kind of very important aspect to redesign an assessment redesign in any any exams or any other form of assessment formative assessment you're going to use they're not useful enough they create a superficial environment which sometimes could be ethically very very dubious as far as the students are concerned so i'm glad that this seems to be a convergence from all three presentations about this this quarter quite this quite strong emphasis on redesign and assessment yeah and i mean and are the only thing that i i didn't for talk about here but we went through a rigorous process and now it's kind of we didn't do so much this year but like when we were redesigning it everybody had had a buddy who had to revere review the paper and then i reviewed them from my school and then we reviewed them at a faculty level and we were saying we're doing this questions and timing so there was a kind of a control process put in place so that order wouldn't say kind of like the sound of this question and i'm going to go with this and um so there was and now the exam the subsequence exams once they have kind of the template of how to do things um it's it's you know it can kind of run itself a little bit but i think that piece of somebody overseeing and not just monica having a mad id and doing this question like so there was the body there was there was me and then there was of the kind of a dean-appointed person at a faculty level going through all the kind of procedures how much time you're doing are you doing mcqs um and like we spotted you know those colleagues were things mad stuff like if well i wrote the question they have 20 questions i'll give them i'll give them 20 minutes to do it like i mean you know it took me five to do it and i'm thinking like you designed the questions you are the experts in the field it would take you at least a minute to read each question and you know if the mcqs are the distractors are sufficiently robust you know so it's kind of um managing um colleagues expectations and the other important thing i think as well was it was really important to give students an exam an example of what the exam would look like so that if they were used to this kind of normal exam paper from last year and using that as a guideline when they moved to this year and it's completely different even just the physical look and feel of it is different they kind of think you know so it was really important to maybe not for every single question but like that example i gave of the the avocados in chile or china or whatever it was just to get in the sense of normally you'd be asked to calculate it but here we are giving you the answer and you can kind of work out which way it is and then you can see for each student yours is avocados minus oranges and the country is spain rather than chile or whatever it is and so that kind of so you're asking the same thing really but it looks superficially different i thought that quality assurance bit that you brought in monica that that bit was really good inez is patiently waiting with our hands up there sorry it is going back to one of the first questions about the the students with disabilities because we i don't know we have a big experience with that we have a many students with disabilities in spain study at net so we have a special unit for supporting their their needs and their demands and i don't have the data about their performance last year but using only information from my course i can say that the assessment rate was 100 so all the students with disabilities having the online exam was good for them they they had a bigger opportunity to undertake exam for mobility issues like the i think it was more linda who mentioned that people who can't go easily to the regional center to their exam um all the students with disabilities do the exams but this year i have it seems that sometimes of vision problems i don't know exactly what type of blind students the platform has some accessibility issues that have to be solved so it doesn't mean that they can't take the exam but they don't feel confident enough there so then these students can can go to the regional center to date exams take some face face to face any student students with disabilities or students that anticipate and how they can have a internet connection problems or any other difficulty in the online platform they can apply to go to the regional center state exam it is a minority but it is still happening so the regional centers are open for that and about the [Music] the project already signed the the assessment method uh i'm happy to see that it's it's very common in all the in all our cases because in in my university there is a debate between some faculty who are claiming to go back as soon as possible to a traditional face-to-face exam and and others including the the center for distance education they are trying to rethink all the assessment and providing some guidelines including this one you just mentioned of giving examples of questions to the students so they don't feel anxious before the exam and also the university is organizing a series of webinars where uned faculty they they share what they have done specifically in their course so in specific areas and with specific types of assessments so they are sharing that and and it's it's really useful to share that those experiences i think we hear when i talking about this it's great a great discussion um so i'm going to i think i'll bring it to a close because we're we've gone 24 minutes over um but listen i'd like to thank the speakers uh monica linda estelianos and inez really interesting and like it's it's strange considering how quite different contexts in some cases and different countries but very similar conversations especially around the academic integrity the timing uh students with disabilities it's it's astonishing in some ways how similar uh some of the experiences are and so thank you very much for your contributions uh thank you very much to those who attended uh great participation via chat and youtube and we look forward to seeing you at our next eden webinar thanks and bye thanks very much bye everybody thank you all bye bye