Benchmarks

of labelled private details removed: names, emails, phone numbers, addresses, card and bank numbers, IDs.

42,090 of 50,187 values across 31,001 documents nothing was tuned on · ai4privacy/pii-masking-200k · CC-BY-4.0

of emails, IBANs, card numbers, phone numbers and vehicle IDs in the same run.

every labelled value of those kinds · Social Security numbers 89% · bank account numbers 70% · ai4privacy/pii-masking-200k · CC-BY-4.0

of the example secrets gitleaks — a widely used open-source secret scanner — tests its own rules against, written the way keys sit in real code and config.

5,780 of 5,963 examples across 215 kinds of secret · gitleaks · MIT

of people's names found, and 96% of what it removed was personal data rather than an ordinary word.

31,001 documents nothing was tuned on · ai4privacy/pii-masking-200k · CC-BY-4.0

of account, policy and reference numbers no pattern recognises, found by reading the words around them.

8,857 documents with such numbers · ai4privacy/pii-masking-200k · CC-BY-4.0

of texts naming a person passed on to the name check, including names that open a sentence and messages typed without capitals.

622 of 637 texts · Microsoft Presidio Research · MIT

of the cases Microsoft Presidio, the most widely used open-source privacy engine, tests its own recognizers with: national ID numbers, tax numbers and phone numbers from dozens of countries.

393 of 540 cases across 60 kinds of personal data · Microsoft Presidio · MIT

of the secrets Yelp's detect-secrets tests itself with, including passwords compared and assigned in C, C++, Go and config files.

116 of 141 cases · Yelp detect-secrets · Apache-2.0

of the secrets TruffleHog, a widely used secret scanner, tests its 847 detectors with.

1,281 of 1,697 examples · TruffleHog · AGPL-3.0

of the official example phone number for 245 countries and territories, written internationally, and of the example IBAN for all 121 IBAN countries, written unbroken and in groups of four. Local-format numbers are caught every time after words like "call me on", or as the answer to "what's the best number to reach you on?".

490 of 490 international phone numbers · 242 of 242 IBANs · Google libphonenumber metadata · SWIFT IBAN Registry

of the example postcodes Google keeps for 169 countries, written inside an address the way each country lays one out. When the text says what it is ("Postcode: …"), 99.5%.

409 of 424 postcodes in an address · 422 of 424 when labelled · 162 of 169 countries found every time · Google libaddressinput · Apache-2.0

of the example identity numbers python-stdnum tests itself on, when a sentence hands them over ("My ID number is …"): from Brazil's CPF and China's resident ID to Ireland's PPS and Portugal's Cartão de Cidadão.

1,013 of 1,022 numbers across 43 formats · python-stdnum · LGPL-2.1

of labelled values mostly hidden in Microsoft's synthetic dataset, which writes addresses as whole multi-line blocks in dozens of countries' formats.

878 of 968 values · street addresses 89% · cards, phones, emails, IBANs and SSNs all 100% · Microsoft Presidio Research · MIT

findings on anything the annotators had not marked as identifying, across every judgment in the corpus.

92 findings in 1,268 judgments · 9.6 million characters · Text Anonymization Benchmark (ECHR judgments) · MIT

fewer false alarms on phone numbers, cards and postcodes in Python's and Kubernetes' documentation: the hex dumps, timestamps, version numbers and config files developers paste.

1,891 false alarms before, 71 now, in 28 million characters · Python documentation · PSF License; Kubernetes documentation · CC BY 4.0

pieces of personal data found in ordinary work email. Two in five messages carried at least one — including one employee's card number, written out in full.

7,100 messages · 38.7% carried personal data · Enron email corpus · public since 2003 (FERC release)

of regulated values stopped in a trace of agent tool calls — emails, requests and database queries an AI agent tried to send.

31 of 32 values across 8 actions · Written by us

of values caught across six batches written without looking at the rules — international formats, whole documents, and data disguised to get past a filter. Ordinary text was left alone 99% of the time.

240 of 246 values · 148 of 150 ordinary sentences untouched · Written by us