Can Pangram’s AI text detector really be trusted?

Pangram has become one of the most visible tools for detecting AI-generated writing, but the number it produces is not enough to settle every dispute. The Brooklyn startup, which has only 24 employees, estimates the share of AI involvement in a text, while its accuracy remains contested.

Max Spero and Bradley Emi founded the company in 2023 under the name Checkfor.ai. It changed its name the following year and has raised about $13 million. Pangram drew major attention after Mia Ballard’s novel “Shy Girl” was suspected of using AI. Although Ballard denied it, Spero said Pangram rated 78% of the manuscript as AI-generated. Hachette subsequently canceled plans to publish the book.

Other works have also received striking scores. An article in The New York Times’ Modern Love section and a Commonwealth Short Story Prize-winning work both received scores of 100%, while the novel “Daggermouth” received 60%. Substack announced in late July that it would integrate Pangram so readers could check the likelihood that content was AI-generated.

The tool’s growing use has prompted concern among writers and publishing professionals. Author and publishing specialist Jane Friedman said many writers fear detection software could cause harm when it produces an incorrect result. Pangram says it trained its system by comparing human-written text with AI-generated content, but the company acknowledges that the tool performs poorly on passages under 100 words. The same text can also receive different scores depending on the test.

A University of Notre Dame study found that Pangram 3.2 incorrectly marked lightly AI-edited text as AI-generated in 64% to 80% of cases. When AI-written text was processed through a “humanizing” tool, Pangram detected fewer than 4% of cases.

Accuracy is not the only concern. Sam Illingworth, a professor at Edinburgh Napier University, warned that AI detectors may contain biases. Research suggests non-native English writers face a higher risk of being misclassified. The article said the cases involving “Shy Girl,” “Daggermouth” and “Call Me, I’ll Hide the Body” all involved authors of color, increasing concerns about unequal effects. Regina Brooks, president of the American Association of Literary Agents, said the publishing industry must consider the inequalities these tools could create. Penguin Random House said editors may use AI detectors, but their results are not treated as sole evidence.

“

Shy Girl