AI Detectors and Non-Native English Writers: What the Stanford Study Actually Found
If English is not your first language and an AI detector flagged an essay you wrote yourself, there is published research you should know about. It does not prove anything about your specific paper, but it explains why a flag on careful second-language writing deserves a closer look before anyone draws conclusions.
The study: Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou, "GPT detectors are biased against non-native English writers," published in Patterns in July 2023 (preprint on arXiv). The authors are researchers at Stanford University.
What the researchers tested
The team ran two sets of human-written essays through seven AI detectors that were widely available at the time, including GPTZero, ZeroGPT, Originality.AI, and OpenAI's own classifier:
- 91 TOEFL essays written by non-native English speakers, collected from a Chinese educational forum.
- 88 essays by US eighth-grade students, from the Hewlett Foundation's ASAP dataset.
Every essay was written by a person, so any "AI-generated" label was a false positive.
What they found
- Across the seven detectors, the average false-positive rate on the TOEFL essays was about 61%.
- The US eighth-grade essays were classified far more accurately, with an average misclassification rate of about 5%.
- 18 of the 91 TOEFL essays (about 20%) were labeled AI-generated by all seven detectors, and 89 of 91 (about 98%) were labeled AI-generated by at least one.
The researchers then had ChatGPT enhance the vocabulary of the TOEFL essays so they would sound more like a native speaker. The average false-positive rate dropped to about 12%. Going the other way, simplifying the word choices in the eighth-grade essays raised their misclassification rate to about 57%.
That experiment is the heart of the paper. The detectors were responding to vocabulary, not to who wrote the text. The authors connect this to perplexity, a measure of how predictable text is to a language model. Writers who use common, safe vocabulary produce more predictable text, and the detectors tested treated predictable text as a sign of AI.
Why this happens
When you write in a language you are still mastering, you tend to choose words you are sure of. That is a sensible strategy. You may also stay close to sentence patterns you have practiced. Both habits make text more predictable to a language model, which is the same property many detectors associate with machine-generated text.
The same signal can have two very different causes, and a score alone cannot tell them apart. For a deeper explanation of the underlying measures, see what perplexity and burstiness mean.
What the study does not show
Read the numbers carefully before you rely on them.
- It tested specific 2023 tools. OpenAI has since withdrawn its classifier, and other vendors have updated their models. Turnitin was not among the tools tested.
- The samples were small and specific. 91 TOEFL essays and 88 eighth-grade essays are not a representative sample of all students or all writing.
- Vendors report different results for their own tools. Turnitin published 2023 testing on nearly 2,000 additional English-language-learner writing samples and reported no statistically significant difference in false-positive rates for documents above its 300-word minimum (Turnitin). That is a vendor testing its own product, so weigh it accordingly.
So "61%" is not the error rate of whatever detector your school uses. It is evidence that detectors of this kind can misread second-language writing at high rates, and a good reason to ask for more than a score.
Institutions have noticed
When Vanderbilt University disabled Turnitin's AI detector in August 2023, its announcement cited concerns about false positives and transparency, along with research suggesting detectors are more likely to label non-native English writing as AI-written (Vanderbilt). Turnitin's own guidance says its AI writing score may misidentify text and should not be the sole basis for adverse action against a student (Turnitin guides).
What to do if you have been flagged
1. Respond calmly and in writing. Ask to meet, say that you wrote the work yourself, and offer to share your drafts and notes. You can mention the study politely, for example: "Research by Liang et al. (Patterns, 2023) found that several AI detectors frequently mislabeled essays by non-native English writers as AI-generated. I'd appreciate the chance to show my drafting process."
2. Gather your version history. In Google Docs: File → Version history → See version history. In Word, version history is available for files saved to OneDrive or SharePoint (Microsoft support). Screenshot early and late versions with their timestamps.
3. Collect process evidence. Outlines, handwritten notes, annotated sources, earlier drafts, and messages about the assignment all help show how the work developed.
4. Learn the process. Read your institution's academic-integrity policy. Check whether you can bring an advisor, respond in writing, or appeal. International students can also ask their international student office or the dean of students office what support is available.
5. Take the time you need. If you are asked to sign or admit anything, it is reasonable to ask for the concern in writing and for time to respond. Check your policy for what applies at your institution.
Habits that help for every paper
Draft in a tool that keeps version history automatically. Save your outlines, notes, and annotated sources alongside your drafts. Write your outline and opening paragraphs in your own words before using any tool your course allows, and disclose tool use the way your course requires. None of this is paranoia. It is the same record a researcher keeps.
What instructors can take from the research
The study's authors caution against relying on these detectors in evaluative or educational settings. Practical steps that follow: treat a flag as a reason to talk, not a verdict; ask the student to walk through specific choices in the paper; and consider asking for process evidence, such as version history or an outline, as part of major assignments.
A note on editing
You do not need to change your voice to satisfy a detector, and no tool can promise a detector result. If you want help with phrasing and your course permits AI-assisted editing, HumaText can propose a phrasing pass on a short section of your draft. Review every change to make sure it still says what you meant, and keep your version history first. It is free to try without an account.
If you were flagged for writing you did yourself, the research gives you a reasonable basis to ask for more than a score. Bring your drafts, bring your notes, and ask to discuss the work itself.