AI System Verifies Authorship with 85% Accuracy, Beating ChatGPT Mimicry

Table of Contents

• The Problem: AI-Generated Content Bypassing Detection
• Proposed Solution: AI System Identifying True Authorship Style
• Key Technical Details of the Authorship Verification System
• Results: State-of-the-Art Performance Identifying AI-Generated Text
• Conclusions and Applications for AI Authorship Verification

The Problem: AI-Generated Content Bypassing Detection

The rise of advanced AI systems like ChatGPT has enabled the automatic generation of human-like text. This poses a major problem - AI-generated content can now bypass plagiarism detection systems through simple techniques like paraphrasing and style mimicry. Without effective detection methods, AI-written text can be falsely presented as original human work.

Current plagiarism checkers rely on pattern matching and other superficial signals. They fail to verify the true authorship behind content. AI systems can now study an author's writing style and mimic it with high accuracy. This allows AI-generated text to slip past plagiarism detectors when masquerading as a specific person's content.

Current Systems Fooled by Paraphrasing and Style Mimicry

Most plagiarism detection services in use today match textual patterns to flag duplication. Some use fingerprinting to catch near-duplicates. However, they do not evaluate writing style and cannot authenticate authorship. AI models like GPT-3 and ChatGPT can easily paraphrase content, preserving semantic meaning while altering phrasing to dodge pattern matching. Instructing the AI to mimic a target authoring style allows generating content closely matching that person's writing. This causes current plagiarism systems to fail to detect the AI authorship. Without considering true authorship style, duplicate content detection cannot adapt to these AI obfuscation techniques.

Proposed Solution: AI System Identifying True Authorship Style

To address this flaw, we propose an AI-powered solution focused on verifying authorship style. The key advancement is using machine learning to evaluate writing characteristics specific to an individual. This allows detecting AI-generated content even when paraphrased or disguised as another author.

We train neural networks to extract stylometric embeddings capturing markers of an author's unique writing style. These embeddings are compared to new text to authenticate if it matches the expected authorship. The embeddings focus on stylistic aspects like vocabulary, syntax, and semantics rather than surface patterns.

Key Technical Details of the Authorship Verification System

Our verification system uses neural networks in a novel architecture designed for highly accurate authorship identification and forgery detection. The key components driving the performance gains are:

1. Stylometric text embeddings that quantify markers of an author's distinctive writing style

2. A Siamese neural network architecture optimized for classifying authorship

3. Curriculum learning for improved accuracy, especially on AI-generated content

Stylometric Text Embeddings

We extract a rich set of 58 stylometric feature values from input text, spanning lexical, syntactic, structural, and semantic attributes. This builds a multidimensional representation of writing style. A transformer model condenses these into a compact 256-dimension embedding vector. This becomes a signature of the author's distinctive style.

Siese Neural Network Architecture

A Siamese neural network compares the embedding of the check text to the context embedding of example writings from the claimed author. It contains twin transformer encoders to process each text input. The resulting vectors feed into a classifier predicting match or non-match of authorship style.

Curriculum Learning for Accuracy Gains

We employ curriculum learning which initializes models on easier tasks before fine-tuning on harder ones. First, we pre-train the encoder network on a corpus of general academic writing samples. Then we freeze encoder weights and train the full Siamese network on the Reuters news dataset. Finally, we fine-tune solely the classifier layers on AI-generated texts. This improves accuracy by 4.7% on average.

Results: State-of-the-Art Performance Identifying AI-Generated Text

Our proposed system achieves new state-of-the-art results for authorship verification, significantly outperforming previous benchmarks. The combination of stylometric embeddings and optimized neural architecture proves highly effective.

85% Accuracy Exposing ChatGPT Style Mimicry

When evaluated on our test dataset of ChatGPT outputs imitating author styles, our system correctly flags AI-generated content 85% of the time. This demonstrates high accuracy even on adversarial AI text trying to mimic human writing.

Surpassing Accuracy of Decommissioned OpenAI Classifier

OpenAI previously used a classifier model to detect AI-generated text from GPT-3. However, it only achieved 72% accuracy, prompting its decommissioning. Our proposed architecture surpasses this previous benchmark by over 13 percentage points.

Conclusions and Applications for AI Authorship Verification

This work demonstrates a novel approach using neural networks and stylometric embeddings to reliably verify authorship of text. The proposed system significantly raises the bar for identifying AI-generated content.

It has profound implications across many sectors including education, journalism, governance and more. By exposing AI mimicry and fake content, this technology can protect information integrity and authenticity.

FAQ

Q: How does this AI author verification system work?
A: It analyzes the style of a text through stylometric analysis into a signature, then a neural network compares it against examples of an author's writing.

Q: What accuracy does it achieve in detecting AI-generated text?
A: In testing, it achieved approximately 85% accuracy in identifying when AI like ChatGPT tries to mimic an author's writing style.

Q: How does it perform compared to other AI detection systems?
A: It demonstrated state-of-the-art performance, significantly exceeding the accuracy of OpenAI's now-decommissioned classifier system.

Q: What are some applications for AI authorship verification?
A: It has widespread applications protecting academic integrity, combating disinformation, ensuring authenticity in writing, and more.