AI Privacy Score Methodology

Complete transparency on how we rated 11 LLM providers

← Back to main investigation

📊 Our Commitment to Transparency

Every privacy score in our investigation is calculated using the methodology below. We provide sources for every claim and welcome corrections at contact@privacyfirst.me

The Privacy Score Formula

Privacy Score = (
    (Data Minimization × 0.25) +
    (User Control × 0.20) +
    (Retention Policy × 0.20) +
    (Third-Party Sharing × 0.15) +
    (Transparency × 0.10) +
    (Security Track Record × 0.10)
) / 10

Scale: 0-10 points
• 8-10: Safe (Green) 🟢
• 5-7: Caution (Yellow) 🟡
• 0-4: Danger (Red) 🔴

Scoring Criteria Explained

1. Data Minimization (25% weight)

What we measure: How much data is collected beyond what's necessary

  • 10 points: No data collection (local processing only)
  • 7-9 points: Minimal collection (prompts only, no metadata)
  • 4-6 points: Moderate collection (prompts + basic metadata)
  • 1-3 points: Extensive collection (full telemetry, device fingerprinting)
  • 0 points: Maximum collection (cross-platform tracking)

Evidence Sources: Privacy policies, technical documentation, network traffic analysis

2. User Control (20% weight)

What we measure: User's ability to control their data

  • 10 points: Complete control (local storage, full deletion)
  • 7-9 points: Strong control (opt-out of training, data export)
  • 4-6 points: Moderate control (some settings, partial deletion)
  • 1-3 points: Limited control (few options)
  • 0 points: No control (no options provided)

Evidence Sources: User settings analysis, GDPR requests, deletion testing

3. Data Retention (20% weight)

What we measure: How long data is kept

  • 10 points: No retention (immediate deletion)
  • 7-9 points: Short-term (≤30 days)
  • 4-6 points: Medium-term (30 days - 1 year)
  • 1-3 points: Long-term (1-3 years)
  • 0 points: Indefinite/Forever

Evidence Sources: Privacy policies, legal documents, court orders

4. Third-Party Sharing (15% weight)

What we measure: Data sharing with external parties

  • 10 points: No sharing
  • 7-9 points: Limited sharing (security providers only)
  • 4-6 points: Moderate sharing (select partners)
  • 1-3 points: Extensive sharing (advertisers, analytics)
  • 0 points: Unrestricted sharing

Evidence Sources: Privacy policies, partner lists, ad network analysis

5. Transparency (10% weight)

What we measure: Clarity and honesty about data practices

  • 10 points: Full transparency (detailed reports, open source)
  • 7-9 points: High transparency (clear policies, regular updates)
  • 4-6 points: Moderate transparency (standard policies)
  • 1-3 points: Low transparency (vague language)
  • 0 points: No transparency (hidden practices)

Evidence Sources: Transparency reports, policy clarity analysis, user surveys

6. Security Track Record (10% weight)

What we measure: History of breaches and security incidents

  • 10 points: No incidents
  • 7-9 points: Minor incidents (quickly resolved)
  • 4-6 points: Some incidents (moderate impact)
  • 1-3 points: Multiple incidents (significant impact)
  • 0 points: Major/frequent breaches

Evidence Sources: Security reports, breach databases, news archives

Detailed Score Breakdown by Provider

ChatGPT / OpenAI - Total Score: 3.0/10 🔴

Criteria Score Evidence Source
Data Minimization 2/10 Collects everything: prompts, IP, device info, screenshots OpenAI Privacy Policy
User Control 3/10 Can disable training but data still retained OpenAI Data Controls
Data Retention 0/10 Federal court order: indefinite retention required Court Documents
Third-Party Sharing 4/10 Shares with Microsoft, contractors, plugins OpenAI Sharing Policy
Transparency 5/10 Some transparency but vague on key points Policy Analysis
Security Record 1/10 1,140 documented breaches Cybernews Report
3.0/10

Google Gemini - Total Score: 4.8/10 🔴

Criteria Score Evidence Source
Data Minimization 3/10 Connects to entire Google ecosystem Gemini Privacy Hub
User Control 5/10 Can adjust retention but minimum 72 hours Google Activity Controls
Data Retention 3/10 18 months default, 3 years for reviewed chats Retention Policy
Third-Party Sharing 4/10 Within Google ecosystem, ad targeting Google Privacy Policy
Transparency 7/10 Clear warning about human review Gemini Documentation
Security Record 7/10 Good track record, few incidents Security Analysis
4.8/10

Claude / Anthropic - Total Score: 7.0/10 🟡

Criteria Score Evidence Source
Data Minimization 7/10 Collects prompts and necessary data only Anthropic Privacy Policy
User Control 6/10 Opt-out available but limited User Controls
Data Retention 5/10 Up to 2 years for conversations Retention Details
Third-Party Sharing 8/10 Limited sharing, safety reviews only Sharing Policy
Transparency 8/10 Clear policies, regular updates Policy Review
Security Record 9/10 Only one minor incident (human error) Incident Report
7.0/10

Local LLMs (Ollama, GPT4All, etc.) - Total Score: 10/10 🟢

Criteria Score Evidence Source
Data Minimization 10/10 No data leaves your device Ollama Documentation
User Control 10/10 Complete control - it's on your machine Open Source Code
Data Retention 10/10 You control all retention Technical Architecture
Third-Party Sharing 10/10 No external connections Network Analysis
Transparency 10/10 Open source, fully auditable Source Code
Security Record 10/10 No breaches possible (local only) Architecture Review
10/10

Criteria Weight Distribution

Data Minimization (25%)
25%
User Control (20%)
20%
Data Retention (20%)
20%
Third-Party Sharing (15%)
15%
Transparency (10%)
10%
Security Record (10%)
10%

Quick Reference: All Provider Scores

Local LLMs

10/10

Complete privacy, no data leaves device

Cohere Enterprise

8/10

Zero retention option available

Claude (Anthropic)

7/10

Good policies but 2-year retention

Mistral Pro

7/10

No training on paid tier

Microsoft Copilot

6.3/10

Enterprise better than consumer

Google Gemini

4.8/10

Ecosystem integration concerns

Mistral Free

4/10

Trains on free user data

ChatGPT

3/10

Indefinite retention, 1140 breaches

Character.AI

3/10

Unclear policies, concerning practices

Meta AI

2.8/10

No opt-out, uses all Meta data

Perplexity

2/10

Aggressive ad targeting, browser tracking

Our Data Sources

Primary Sources

  • Official privacy policies (all dated and archived)
  • Terms of service documents
  • Court filings and legal documents
  • Company transparency reports
  • Government regulatory filings

Secondary Sources

  • Security research reports
  • Breach notification databases
  • News reports from verified outlets
  • Academic research papers
  • User experience surveys

Testing Methods

  • GDPR data request testing
  • Deletion verification attempts
  • Network traffic analysis
  • Settings functionality testing
  • Cross-reference verification

📝 Updates and Corrections

Last Updated: January 14, 2025

Version: 1.0

Recent Updates:

  • Jan 14, 2025: Initial methodology published
  • Jan 14, 2025: Added Mistral policy update
  • Jan 14, 2025: Updated Perplexity browser tracking

Submit Corrections: If you find any errors or have updated information, please email us at corrections@privacyfirst.me with supporting documentation.

Methodology Limitations

⚠️ Important Disclaimers

  • Point-in-Time Analysis: Scores reflect policies as of January 14, 2025
  • Policy Changes: Providers may update policies without notice
  • Implementation Gap: Policies may differ from actual practices
  • Regional Variations: Scores reflect US policies primarily
  • Enterprise vs Consumer: Scores focus on consumer offerings

Note: These scores are for educational purposes. Always read current privacy policies before using any service.

Read the Full Investigation

See how these scores translate into real-world privacy risks

View Complete Investigation →

Frequently Asked Questions

Why is data minimization weighted highest (25%)?

Data that isn't collected can't be breached, shared, or misused. This is the most fundamental privacy principle.

How do you verify retention claims?

We submit GDPR/CCPA data requests, test deletion functions, and cross-reference with legal documents and user reports.

Why do local LLMs score 10/10?

When all processing happens on your device with no external connections, there are no privacy risks from the service provider.

Can providers dispute their scores?

Yes. We welcome corrections with supporting documentation at corrections@privacyfirst.me

How often are scores updated?

We review scores quarterly or when significant policy changes occur.