AI translates Swedish better than Hindi: what matters is where the labs focus their effort

A team from Smartling, an AI translation platform, tested how the latest models from the leading frontier labs translate seven languages: Estonian, Hindi, Indonesian, Brazilian Portuguese, Swedish, Thai and Turkish. The analysis is by Olga Beregovaya, VP of AI at Smartling, as a contributed piece for Forward Future.

The main finding goes against intuition: the number of speakers has almost no bearing on translation quality. Swedish, with about 10 million speakers, got some of the best translations in the study. Hindi, with about 600 million, was among the worst, along with Estonian and Thai. According to the author, a Hindi speaker is served worse than a Swedish speaker even though Hindi has 60 times as many speakers.

Methodology: each model was given the same set of about 500 sentences per language, drawn from four common business domains. The translations were scored with several industry-standard metrics so that no single figure would skew the result. Two model generations from each lab were also tested, to show both where things stand now and how they have changed.

Why it matters: AI is already used in banking, healthcare and education, and anyone who builds these models into a product inherits their blind spots. The question used to be whether a model supported a language. Today models produce text in hundreds of languages, and the question is how good that text is.

Explanations offered in the piece: part of the gap comes from the languages themselves. Brazilian Portuguese has a fairly simple grammar and plenty of available text. Swedish, a relative of German, has shed much of its grammatical complexity. Estonian changes the form of words a great deal depending on their role in the sentence, which models find hard. Hindi lags because models have not been trained on enough Hindi text, or on enough variety of it.

However, grammar and data explain only part of the picture. Translation is just one task among many for the labs, which also work on reasoning, coding, agents and safety, so each release reflects where the effort went. Tokenization (how a model splits text into pieces) matters most for scripts such as Hindi and Thai, and updates to it tend to follow the priorities of the product roadmap.

Progress: the languages that started furthest behind usually improved the most from one generation to the next, though not always, and progress varied from lab to lab. New models also often change formatting, verbosity or punctuation in ways that shift the metrics even when the meaning is preserved. The author argues that the priorities of three or four companies in a given cycle can decide whether a language catches up with the others.

Solutions: the methods are well known: more multilingual text, better tokenizers, synthetic data (text generated by models to fill gaps) and cross-lingual transfer (for example, a model trained on a lot of Russian can extrapolate to Slovenian). What is missing is focus. Labs could put someone in charge of neglected languages and publish per-language results with each release. Companies building on these models can test them in their customers' languages and share the results with the labs. Smartling shares its findings with the labs and works through the causes with their scientists, who, according to the author, are open to it.

Context: the author notes that in September a Gates Foundation coalition brought frontier labs together in a joint five-year commitment to develop AI support for low-resource languages. She sees this as a sign that the labs are starting to treat these languages as a priority. Her conclusion is plain: the gap narrows where the labs put in the work, and Hindi, Estonian and Thai will improve only if labs and companies decide those languages matter.

It is worth noting that this is an opinion piece from a company in the translation industry, based on a study of seven languages with about 500 sentences per language. The piece does not say which models or which specific metrics were used.