Messages sent by the Office of Instructional Technology
The Office of Instructional Technology sent out messages to Faculty and Students in later April of 2023 to try and address this issue as far as we were able to do so from a technical standpoint. The links to those are below:
Considerations when Turnitin Indicates a paper contains AI Writing
If a paper is marked as 100% AI-generated, Turnitin would have high confidence in their assessment that a substantial part of the paper was AI writing, if not the entirety. However, other factors might come into play. Was it a very short sample of writing? Was it a relatively generic topic that might not return a lot of unique viewpoints? How has this student's work been in other parts of the class? Does this writing seem like a departure from other work they've turned in?
Because AI writing is technically original writing there is nothing to check the validity of the writing against, so it's always going to come down to a judgment call and maybe a secondary type of assessment - can the student verbally convey the same ideas as in the paper? Can they show you the specific sources they consulted? Do they have any drafts of their work?
Can Turnitin identify writing edited by AI?
Turnitin representatives said the following regarding AI "writing enhancement" tools such as GrammarlyGo:
The first iteration of Turnitin’s AI writing detection capabilities have been trained to detect models including GPT-3, GPT-3.5, and variants. Our technology can also detect other AI writing tools that are based on these models such as ChatGPT. We’ve completed our testing of GPT-4 (ChatGPT Plus), and the result is that our solution will detect text generated by GPT-4 most of the time. We plan to expand our detection capabilities to other models in the future.
When using an AI-powered grammar check, if changes to the writing are small, our model is unlikely to predict it as AI writing. If the tool makes bigger rephrasing changes, to several sentences in a row, it likely will be flagged by our detector. Our detector looks in a broader way, as in larger blocks of text rather than small improvements in sentences.
We only flag something as AI-written when we are 98% sure it is written by AI, with a less than 1% false positive rate (incorrectly identifying fully human-written text as AI-generated). This is because we want to make sure we don’t falsely flag something as AI-generated that isn’t.
Does Turnitin have plans to build a solution to detect when students paraphrase content either themselves or through tools such as Quillbot?
Turnitin sent us the following:
Turnitin has been working on building paraphrase detection capabilities – the ability to detect when students have paraphrased content either with the help of paraphrasing tools or re-written it themselves – for some time now, and the technology is already producing the desired results in our AI Innovation Lab. In the instance when the student is using a word spinner or an online paraphrasing tool, the student is just running content through a word spinner which uses AI to intentionally subvert similarity detection, not using generative AI tools such as ChatGPT to create content. We have plans for a beta release in 2023, and we will be making paraphrase detection available to instructors at institutions that are using TFS with Originality and Originality for an additional cost. It will be released first in our TFS with Originality product."
Questions?
This is a challenging topic and unfortunately, there is no clear-cut answer on whether you can rely on the Turnitin AI writing detector. Please reach out to the Office of Instructional Technology if you have further questions.