This page contains Blackboard Original support material


Visit Blackboard Ultra Staff support for information and guides for teaching with Blackboard Ultra
Visit Student Blackboard Ultra support for information and guides for digital learning with Blackboard Ultra

Turnitin AI Detection

Owned by Andy Jaffrey
Last updated: Jul 18, 2023
4 min read

Background

In Spring 2023, Ulster established an AI in Learning, Teaching & Assessment Working Group to consider guidance, and policy changes, in response to the increased sector discussions on Artificial Intelligence technologies in Education.

The working group was established in response to increased visibility of Large Language Models (LLMs) following the launch of ChatGPT in November 2022. Media coverage generated both excitement and panic with examples of outputs successfully passing professional exams, writing passable job applications and producing work which could pass many assessments in education.  Academia was ‘stunned by ChatGPT’s essay writing skills and usability’ (Herne 2022), but some in the sector approached the issue from a more critical and nuanced perspective urging us to ‘get off the fear carousel’ (Mihal 2023).

Naturally, many academic colleagues were concerned about academic integrity particularly following their own experimental activity prompting LLMs to iteratively generate responses to assessment briefs. Much of this experimentation resulted in passable submissions and this, combined with media coverage and subject specific conversations, resulted in a demand for institutionally approved detection tools. In the absence of an institutionally approved detection tool some academic colleagues began experimenting with web based detection tools, in some cases paying for their own subscription.

There are ethical issues with these approaches, both in the use of LLMs in the first place but also in the use of detection tools which necessitate uploading students' work to unknown systems without adequate policy, governance or ethical consideration. Many on the working group felt uneasy about the use of detection tools and did not endorse a University approach to the use of tools such as Turnitin AI detection or other tools such as Zero ChatGPT. Experts, from within the University, who teach and research AI, did feel that these tools were easily manipulated by subtle transformation approaches through different tools and they cited examples of how students can ensure their work is not flagged by these tools. Enabling detection tools was described as ‘no win arms race’. However some members did present a case, which was endorsed, that we should experiment, and learn, before making a final decision.

A decision was made to proceed with enabling Turnitin AI detection with the caveat that adequate guidance was provided to ensure that users understood the limitations of detection tools and were aware of ethical issues when using the tools.

Reliability

Since Turnitin launched AI detection there have been concerns about the reliability of the tool, this had prompted a cautious approach in many UK HE institutions where the feature has been disabled whilst a review is completed. It is very difficult to accurately test the tool, at scale, without enabling the feature and Ulster’s working group have made the decision to proceed, with caution.

Jisc and the QAA have provided helpful information on detection tools and urge caution in their use.

Jisc notes: “AI detectors cannot prove conclusively that text was written by AI.” 

Michael Webb (17/3/2023), AI writing detectors – concepts and considerations, Jisc National Centre for AI

The The Quality Assurance Agency for Higher Education (QAA) advises: “Be cautious in your use of tools that claim to detect text generated by AI and advise staff of the institutional position. The output from these tools is unverified and there is evidence that some text generated by AI evades detection. In addition, students may not have given permission to upload their work to these tools or agreed how their data will be stored.”

QAA (31/1/2023), The rise of artificial intelligence software and potential risks for academic integrity: briefing paper for higher education providers

With this in mind, we encourage caution when using the Turnitin AI detection tool whilst we collectively learn about the functionality. As with Originality Scores, within Turnitin, AI detection scores will not definitively prove that a submission has been plagiarised but will provide further information to help with any academic integrity discussion where a submission is of concern.

Turnitin’s support pages (as of July 2023) state that their detection system checks against GPT-3, GPT-3.5 and ChatGPT and that testing has been completed using other LLMs. It is entirely possible for a student to use a variety of LLMs to refine a piece of work and Turnitin provide no reassurance for the reliability of their model when this approach is used. This is yet another reason to treat the AI detection score with caution.

Functionality

AI detection is only available in the latest version of Turnitin (LTI) and details about how to add a new LTI dropbox are available here Turnitin AI Detection | Functionality

Turnitin has further information online at https://www.turnitin.com/solutions/ai-writing which describes the functionality but this is written from a marketing perspective and should be considered critically when understanding the solution. The FAQs page provides useful background information about the feature.

This short instructional video describes the workflow.

An AI detection score is accessible within the Originality Report window as shown below.

When you click the AI detection score you will receive some more information about the text which has been matched as well as additional resources to help with any review.

Further information guidance is available in the attached PDF below

Please note, as of July 2023 students will not see their AI detection score.