Multilingual red teaming for LLMs: why native speakers find what English testing misses
Multilingual red teaming for LLMs means attacking your model in the languages your users actually speak, not only in English, and it works best when the testers are native or near-native speakers. Published research shows that safety behaviour tested in English does not carry over cleanly to other languages: unsafe answers rise when the same harmful request arrives in another language, and human testers who write naturally in a language get through more often than machine-translated prompts. If you ship a model in Spanish, French, German or Japanese, you need people who can probe it in those languages, with the cultural judgment to know what "harmful" looks like there.
In short
- English-only red teaming leaves gaps. Studies found unsafe-answer rates several times higher in non-English languages than in English.
- High-resource languages are not immune. Spanish, French, German and Japanese are safer than rare languages, but still not as safe as English in testing.
- Native speakers beat machine translation in multi-turn, adaptive attacks, where phrasing and cultural context matter most.
- The market already hires for this: AI safety listings ask for native or near-native fluency in a specific language.
- Start small: a 2–5 person pilot in one or two languages, with a clear taxonomy and reporting format.
What multilingual red teaming for LLMs actually covers
Red teaming is structured adversarial testing: people try to make a model produce harmful, unsafe or policy-breaking output, document what worked, and feed the findings back into training and safety filters. Multilingual red teaming does the same thing in each language you support. In practice that includes:
- Direct attacks written natively in the target language, not translated from an English list.
- Multi-turn escalation, where a tester builds up a harmless-looking conversation before the real request.
- Code-switching, mixing English with Spanish, French, German or Japanese in one prompt to slip past filters.
- Culturally specific harms: slurs, euphemisms, local legal topics and stereotypes that an English-speaking reviewer would not recognise.
- Classification and write-up: labelling each finding against your harm taxonomy and recording reproducible steps.
The job is closer to LLM evaluation than to translation. If you are also building an evaluation team, our guide on how to hire multilingual LLM evaluators covers the overlapping skills and tests.
What the research says about non-English jailbreaks
Three published studies make the case clearly. They tested earlier models, and today's systems have improved, but the pattern is the reason to measure safety per language rather than assume it.
| Study | What was tested | Key finding |
|---|---|---|
| Deng et al., Multilingual Jailbreak Challenges in LLMs | Harmful prompts in English and nine other languages, on ChatGPT and GPT-4 | Low-resource languages showed about three times the likelihood of harmful content compared with high-resource languages. With a malicious instruction added, unsafe output reached 80.92% for ChatGPT and 40.71% for GPT-4. |
| Yong et al., Low-Resource Languages Jailbreak GPT-4 | 520 harmful prompts translated into 12 languages | Attack success was 0.96% in English, 10.96% when four high-resource languages were combined, and 79.04% when four low-resource languages were combined. |
| Marx and Dunaiski, Multilingual jailbreaking using low-resource languages | Human red teamers vs. automatically translated prompts in English and four African languages | Human red teaming produced harmful responses 75.8% of the time on average, against 59.8% for automatically translated prompts. The authors found translation quality was the critical factor. |
Two points matter for teams shipping in European languages and Japanese. First, the gap is not only about rare languages. In Deng et al.'s results, ChatGPT gave unsafe answers to 0.63% of plain English prompts but 4.34% on average across the high-resource languages tested. Second, the Yong et al. authors warn that "red-teaming LLMs solely on monolingual, high-resource settings will create the illusion of safety", and urge that red-teaming results be reported beyond English.
Why native speakers, not machine translation
It is tempting to translate an English attack list and run it in every language. For simple, single prompts that can be a reasonable baseline: Deng et al. found machine-translated prompts produced unsafe content at a similar rate to human translations (11.15% vs. 10.19%). The gap opens when testing becomes realistic. In Marx and Dunaiski's study, people who wrote and adapted attacks themselves across multi-turn conversations beat automatic translation in every language they tested, and single-turn translated attacks were largely ineffective.
Native speakers add three things a translation pipeline cannot:
- Natural phrasing. Stiff, translated-sounding prompts are easier for a model to flag. A native writer sounds like a real user.
- Cultural context. As one AI data firm puts it, effective multilingual safety "requires native-speaker experts who understand both the language and the cultural context". Euphemisms, regional slang and local sensitivities do not translate.
- Judgment on the output. Spotting that a Japanese answer is subtly offensive, or that a German reply gives dangerous advice in polite language, takes real fluency.
What to look for when you hire multilingual red teamers
The market already treats native fluency as the entry ticket. Recent AI safety listings for Spanish and Japanese generalist experts ask for "native or near-native fluency" plus business-level written English, and describe the work as writing prompts on sensitive subjects, classifying conversations against guidelines and flagging adversarial phrasings. A red-teaming specialist listing from LILT asks for a native speaker of one of its listed languages, which include Spanish, German, French and Japanese.
| Skill | Why it matters | How to check it |
|---|---|---|
| Native or near-native writing (C1+ or JLPT N1) | Attacks must read like real users; output judgment needs nuance | Short paid writing and rating task on your own guideline |
| Strong written English | Findings are usually documented in English for your safety team | Review a sample write-up |
| Cultural and current-affairs knowledge | Finds harms specific to a country or region | Ask for examples of locally sensitive topics and how a model might fail |
| Structured thinking | Findings must map to your harm taxonomy and be reproducible | Classification exercise against your categories |
| Resilience and clear boundaries | The work involves disturbing content | Discuss content exposure, breaks and opt-outs up front |
Certificates are a starting point, not proof. Our explainer on what CEFR and JLPT levels mean on the job maps each level to the work it supports.
How to set up a multilingual red teaming programme
- Pick languages by exposure. Start with the languages where you have the most users or the highest regulatory risk.
- Share one harm taxonomy across languages, with room for language-specific categories that testers propose.
- Mix attack styles: native single prompts, multi-turn escalation, code-switching and role-play.
- Standardise reporting: prompt, model response, category, severity and steps to reproduce, written in English with the original text attached.
- Measure per language: valid findings per hour, duplicate rate and severity mix, so you can compare languages fairly.
- Protect testers and data: named accounts, least-privilege access, no local copies, and agreed limits on exposure to harmful content.
Our guide to running a 2–5 person pilot of a remote multilingual team from India covers the timeline, time zones and scorecard in more detail.
Multilingual red teaming with Fluentras
Fluentras deploys Indian Spanish, French, German and Japanese speakers to work remotely from India inside your own team, in your tools and under your guidelines. AI data work is our lead area: LLM evaluation, annotation, prompt writing, language QA and red-teaming, with business operations and healthcare roles as well. We test each person's language level ourselves before you see a profile. A typical start is a pilot of two to five people, billed hourly, with a shortlist in about five working days. Our team already supplies remote talent from India to large enterprise teams.
Need red teamers in Spanish, French, German or Japanese?
Tell us the languages, levels and harm areas you want covered, and we will send tested profiles for a 2–5 person pilot in about five working days.
Frequently asked questions
What is multilingual red teaming for LLMs?
It is adversarial safety testing of a language model in each language it supports. Testers try to make the model produce harmful or policy-breaking output in that language, then classify and document what worked so the model can be fixed.
Is English red teaming enough if my model is also used in Spanish, French, German or Japanese?
No. Published studies found unsafe-answer rates rise when the same requests are made in languages other than English, including high-resource languages. Safety needs to be measured in each language you ship.
Can we just machine-translate our English attack prompts?
It is a useful baseline for simple single prompts, but research found human testers writing natively in multi-turn conversations got harmful responses more often than automatically translated prompts. Native testers also catch culturally specific harms that translation misses.
What language level do multilingual red teamers need?
Native or near-native writing, which usually means CEFR C1 or above, or JLPT N1 for Japanese, plus strong written English for reporting. Test it with a short task on your own guideline rather than relying on certificates alone.
How do we start a multilingual red teaming pilot?
Pick one or two languages, agree a harm taxonomy and reporting format, and run a pilot of two to five testers for a few weeks, measuring valid findings per hour, duplicates and severity for each language.
Sources
- Deng, Zhang, Pan and Bing, Multilingual Jailbreak Challenges in Large Language Models (arXiv)
- Yong, Menghini and Bach, Low-Resource Languages Jailbreak GPT-4 (arXiv)
- Marx and Dunaiski, Multilingual jailbreaking of LLMs using low-resource languages (arXiv, 2026)
- Careerflow, Why AI Safety Requires Multilingual Human Experts
- RemoWork, Bilingual Spanish Generalist Expert — AI Safety
- RemoWork, Bilingual Japanese Generalist Expert — AI Safety
- Jobgether, Red Teaming Specialist – AI Training (LILT)