The hype has abated so what now for generative AI and assessment?
Jason Lodge, Sarah Howard, Margaret Bearman and Phill Dawson
The beginning of 2023 brought considerable hype and concern about the emergence of generative artificial intelligence (AI), particularly ChatGPT. Much has been written and spoken about the implications for higher education. However, responses continue to iterate as we work towards a better understanding of what to do about this disruptive technology.
There have been many op-eds and blog posts, countless webinars, roundtables, panel sessions and a summit or two. Most have been helpful in making sense of what we are dealing with. However, with all this activity, it is understandable that hype has given way to a mixture of complacency and boredom in some quarters. While the conversation for others again has moved on to equally pressing matters, such as the Universities Accord, generative AI remains a considerable problem.
Particularly, the threat of students employing generative AI to complete assessment tasks continues to evolve in complexity given the countless ways these tools may be used in learning activities. The dynamism of the situation leaves students without a clear sense of what AI use is appropriate or inappropriate to support their learning and future work.
For educators, the core issue persists, that students can easily circumvent the learning process and potentially pass assessment tasks using generative AI. This is a serious threat to assurance of learning as our ability to trust student submissions as a fair and accurate representation of what they know and can do is greatly diminished.
What has become patently evident, since the early days of the Australasian academic year, is that ignoring the problem is not an option. Further, banning generative AI is also not feasible because detecting its use in assessment submissions is difficult, nearing impossible. There remains an urgent need to rethink and/or reform assessment in higher education.
To address this complex and ongoing issue, the Tertiary Education Quality and Standards Agency (TEQSA) supported and worked with us to bring together a group of higher education experts in assessment, academic integrity, and AI in August this year. Over an intense two days, we debated a direction forward to support the sector to begin addressing this critical challenge.
Our aim was to set a course for where we need to head from here. We worked on developing a compass to provide a collective sense of where higher education institutions could go together. The development of generative AI technologies and tools is occurring far too rapidly to provide any prescriptive guidance. In other words, there is no way to develop a complete map of the terrain. Rather, the sector needs to collaboratively traverse the generative AI landscape to adapt and respond. We hope our work will prove useful in setting the direction of travel.
The result is a resource released recently. Assessment Reform for the Age of Artificial Intelligence is a collective effort by this group to move the discussion about generative AI in higher education forward. In addition to the 18 people who worked to create the document, over 40 esteemed leaders, practitioners, and researchers from around the country generously engaged in feedback on what was produced. We provide here a brief overview of our agreed-upon priorities.
Building on previous calls for enhancing assessment practices (particularly David Boud and Associates’ Assessment 2020), we call for a re-examination of what should be assessed, and consequently, how students should be taught. Crucial capabilities include not just proficiency in using AI tools, but a broader comprehension of the ethical, technical, and social implications of AI.
We also contend that assessment designs should adopt a more programmatic approach that extends across a student’s academic program, rather than focusing solely on individual tasks. This programmatic approach makes room for multiple methods, fewer but more integrated tasks, and meaningful feedback between educators and students.
Tracking student progress over time should also become a priority. Through this, educators can gain a more reliable and comprehensive picture of student learning. This shift in focus places greater emphasis on the learning process, allowing for assessments that require thinking, decision-making, and ethical judgement. AI’s limitations in simulating these human qualities are acknowledged, and assessments should be structured to reveal these aspects.
The use of AI in educational settings should also be viewed through lenses of collaboration and inclusivity. Students should be taught how to collaborate responsibly with both peers and AI, with clear guidelines on what is permissible. While collaboration is encouraged, assessment designs should also be inclusive, ensuring that all students have equal access to AI resources and understand how to use them.
Finally, in terms of security, while it is resource-intensive to secure all assessments against AI interference, identifying key assessments related to program-level outcomes allows for a more focused application of resources. This selective approach ensures that those who complete the program have genuinely done the work and achieved the targeted learning outcomes.
The ideas developed through the TEQSA Assessment Experts Forum are intended to progress the conversation. But they are not the final word. Figuring out how to adapt to this new reality will require ongoing collective effort across the sector. We look forward to continuing the conversation over the months and years ahead beginning with the fifth instalment of the TEQSA and CRADLE webinar series which was held in September. A recording of the webinar is available here:
Acknowledgement: We acknowledge and thank our brilliant colleagues who contributed to and provided feedback on Assessment Reform for the Age of Artificial Intelligence. We also acknowledge and thank TEQSA, particularly Dr Helen Gniel, who has supported and partnered with us on this critical work.
Jason Lodge is Associate Professor of Educational Psychology in the School of Education and a Deputy Associate Dean (Academic) in the Faculty of Humanities, Arts and Social Sciences at The University of Queensland.
Sarah Howard is an Associate Professor of Digital Technologies in Education and Education Lead in SMART Infrastructure at the University of Wollongong.
Margaret Bearman is a Research Professor in the Centre for Research in Assessment and Digital Learning at Deakin University.
Phillip Dawson is a professor and the Co-Director of the Centre for Research in Assessment and Digital Learning at Deakin University