...
The working group was established in response to increased visibility of Large Language Models (LLMs) following the launch of ChatGPT in November 2022. The service reached 100 million users within 2 months of launch. Media coverage generated both excitement and panic with examples of outputs successfully passing professional exams, writing passable job applications and producing work which could pass many assessments in education. Academia has been was ‘stunned by ChatGPT’s essay writing skills and usability’ (Herne 2022), but some in the sector have approached the issue from a more critical and nuanced perspective urging us to ‘get off the fear carousel’ (Mihal 2023).
Naturally, many academic colleagues were concerned about academic integrity particularly following their own experimental activity prompting LLMs to iteratively generate responses to assessment briefs. Much of this experimentation resulted in passable submissions and this, combined with media coverage and subject specific conversations, resulted in a demand for institutionally approved detection tools. In the absence of an institutionally approved detection tool some academic colleagues began experimenting with web based detection tools, in some cases paying for their own subscription.
There are ethical issues with these approaches, both in the use of LLMs in the first place but also in the use of detection tools which necessitate uploading students' work to unknown systems without adequate policy, governance or ethical consideration. Many on the working group felt uneasy about the use of detection tools and did not endorse a University approach to the use of tools such as Turnitin AI detection or other tools such as Zero ChatGPT. Experts, from within the University, who teach and research AI, did feel that these tools were easily manipulated by subtle transformation approaches through different tools and they cited examples of how students can ensure their work is not flagged by these tools. Enabling detection tools was described as ‘no win arms race’. However some members did present a case, which was endorsed, that we should experiment, and learn, before making a final decision.
A decision was made to proceed with enabling Turnitin AI detection with the caveat that adequate guidance was provided to ensure that users understood the limitations of detection tools and were aware of ethical issues when using the tools.
...
With this in mind, we encourage caution when using the Turnitin AI detection tool whilst we collectively learn about the functionality. As with Originality Scores, within Turnitin, AI detection scores will not definitively prove that a submission has been plagiarised but will provide further information to help with any academic integrity discussion where a submission is of concern.
Turnitin’s support pages (as of July 2023) state that their detection system checks against GPT-3, GPT-3.5 and ChatGPT and that testing has been completed using other LLMs. It is entirely possible for a student to use a variety of LLMs to refine a piece of work and Turnitin provide no reassurance for the reliability of their model when this approach is used. This is yet another reason to treat the AI detection score with caution.
Functionality
AI detection is only available in the latest version of Turnitin (LTI) and details about how to add a new LTI dropbox are available here https://ulster.atlassian.net/wiki/spaces/BLS/pages/2165374977/Turnitin+AI+Detection#Functionality
Turnitin has further information online at https://www.turnitin.com/solutions/ai-writing which describes the functionality but this is written from a marketing perspective and should be considered critically when understanding the solution. The FAQs page provides useful background information about the feature.
This short instructional video describes the workflow.
| Iframe | ||||||
|---|---|---|---|---|---|---|
|
An AI detection score is accessible within the Originality Report window as shown below.
...
When you click the AI detection score you will receive some more information about the text which has been matched as well as additional resources to help with any review.
...
Further information guidance is available in the attached PDF below
|
Please note, as of July 2023 students will not see their AI detection score.