What happened
01
In October 2023, Instagram automatically translated some user bios written partly in Arabic.
02
Bios with the word Palestinian, a Palestinian flag emoji and the phrase Alhamdulillah came out as text that included the word ‘terrorist’.
03
The error became widely known through a user's TikTok video.
04
Meta apologized, blamed a bug in its auto-translation and fixed it. The phrase now translates as ‘Thank God’.
Why this is a design failure
The translation put a loaded label into people's own words about themselves. Readers who could not read Arabic had no way to tell that the meaning had changed.
The fair alternative
Translate what people wrote, keep the original in view, and fix errors fast.
Do
Show the original next to the translation and let people report a wrong one.
Don’t
Don’t let a translation add charged words that are not in the original.
Design guide for this case
If you're designing an automatic translation feature
Interface
Show the original text beside every translation, with a "Translated automatically" label and a one-tap "Report wrong translation" button.
Words
In the translation rules: "Never add a charged or negative word that is not in the original. When unsure, translate literally."
Flow
Route reported translations of names, bios and religious or political phrases to human review fast, and hide the translation until it is checked.
Metrics
Don't count number of translations served as success. Track reported errors, especially new loaded words that appear only in the output.
Testing
Run a test set of bios with flags, emoji and common religious phrases in several languages and check every output for added labels.
Before launch
01
Can readers always see the original next to the translation?
02
Is there a quick way to report a wrong translation?
03
Do we block outputs that add charged words missing from the original?