Adept: AI-Generated Text Detection Based on Phrasal Category N-Grams

With the advent of large language models (LLMs), the generation of artificial text has become remarkably accessible and is increasingly integrated into everyday applications. As the use of LLMs to produce content becomes more widespread, the ability to distinguish between AI-generated and human-written texts has grown in importance. This year’s PAN competition focuses on this specific challenge: Based on a text, participants must determine whether it was written by a human or generated by an AI system (more specifically, an LLM). We propose a classification approach called Adept, which explicitly leverages constituent trees to model the grammatical structure of texts. For each sentence, we generate a constituent tree and represent the entire text by aggregating the distribution of syntactic n-grams, defined as paths of a fixed length within these trees. Using these structural representations, we train a multilayer perceptron (MLP) to classify authorship. Adept achieves a mean score of 0.843 on the test dataset, evaluated by the organizers of the competition. This ranks us on rank 16 out of 24 with a score difference of 0.056 to the first place and 0.036 to the third place.