Governments Shape AI Chatbots by Controlling Web Training Data

Ask an AI model the same political question in two different languages, and you may get two very different responses. This isn't a bug in the system. It's a feature of how large language models actually learn.

A new study published in Nature reveals that governments can indirectly influence AI chatbot behavior by shaping the online media environment those systems consume during training. The research team, spanning the University of Oregon, Purdue University, University of California San Diego, New York University, and Princeton University, traced the pathway from state-coordinated media to training data to model output.

According to the Tech Xplore report, the researchers combined evidence from evaluating LLMs in the local languages of 37 countries with a detailed case study from China. Across six studies, they documented how institutional influence leaves measurable traces in what models say.

"People often talk about AI as if it learns from the internet in some neutral way," said Hannah Waight, co-first author of the study and Assistant Professor of Sociology at University of Oregon. "It doesn't. It learns from information environments that have already been shaped by institutions and power, and those environments can leave measurable traces in what models say."

The mechanism is straightforward but insidious. State-coordinated media appears frequently in real training data. When the researchers compared two sources of Chinese state-coordinated media with a major open-source multilingual training dataset derived from Common Crawl, they found more than 3.1 million Chinese-language documents with substantial phrasing overlap. That represents about 1.64% of the dataset's Chinese-language subset.

That is over 40 times the rate for documents from Chinese-language Wikipedia, a common training source. Among documents mentioning Chinese political leaders or institutions, the share rose as high as 23%. Only about 12% of the matched documents came from known government or news domains, suggesting the material had spread widely across the web before reaching AI training corpora.

Brandon M. Stewart, the paper's corresponding author and Associate Professor of Sociology at Princeton University, noted the laundering effect: "State-coordinated content is not just about what appears in official media. It is also about recirculation; the same phrasing moving through newspapers, apps, reposts and ordinary web pages until it looks like part of the broader information environment. Once state-coordinated content is in the training data, the model can launder it into what looks and sounds like neutral, objective information."

The team then tested whether that content could actually shift a model's behavior. Large commercial models take months and millions of dollars in compute to train, so the team experimented with taking a small, open model and adding additional documents to the training process. The results were clear: adding scripted news to the training data made the models more likely to produce more favorable answers—nearly 80% of the time compared with an unmodified model.

This is true even when compared to other non-scripted Chinese media, and especially compared to just adding general Chinese-language text from the internet. When you type a prompt into a chatbot, you're not just getting an answer. You're getting an answer filtered through whatever training data the model consumed, which itself was filtered through whatever media environment existed when that data was scraped.

Eddie Yang, co-first author of the study and Assistant Professor of Political Science at Purdue University, explained: "When the same political question produces systematically different answers with only small changes to the training data, that suggests those additional documents are doing real work."

The language gap reveals the political bias. The team reasoned that if states have strong real-world influence over the pretraining data, it should appear most clearly in the state's primary language. For example, a question about the Chinese government should produce a more pro-government answer when posed in Chinese than the same question posed in English.

They used this within-model, cross-language comparison to probe commercial models without access to their internal parameters. In responses to political questions about China, human raters judged the Chinese-prompted answer to be more favorable to China 75.3% of the time. For prompts not about China, the rate was no different from chance.

The language difference gave them a rare window into a closed system. Follow-on studies using real user prompts and additional commercial models found the same general tendency: on questions about Chinese leaders and institutions, answers tended to be more favorable when the prompt was in Chinese than when the prompt was in English.

This is not just about China. In a cross-national study of 37 countries where a national language is largely concentrated within a single country, models portrayed governments and institutions from countries with stronger media control more favorably in that country's language than in English. The authors emphasize that this result is correlational, but say it is consistent with the mechanism identified in the China case study.

Joshua Tucker, co-author and co-Director of the NYU Center for Social Media, AI, and Politics, added: "The public debate has focused on what AI can generate, but this study points upstream. Before AI systems can influence politics, politics can influence AI."

The physical reality of this matters. When you sit at your keyboard and type a question about a government policy, you're not interacting with a neutral oracle. You're interacting with a statistical model that has memorized patterns from web pages that were themselves shaped by institutional power. The friction you feel when a chatbot refuses to answer certain questions or hedges on sensitive topics isn't just safety filtering. It's the training data speaking.

Commercial models memorized distinctive phrases associated with this material, suggesting they had been seen a number of times during training. The researchers call this idea "institutional influence." It's not about AI companies setting out to curry favor with governments. It's about the training data reflecting the information environment that exists in each country.

The implications for developers are stark. If you're building AI systems for global deployment, you need to understand that your model's behavior will vary by language in ways that reflect the media environments of those languages. This isn't something you can patch with better safety filters. It's baked into the pretraining process.

For users, the takeaway is equally blunt. When you ask an AI a political question, the answer you get depends on what language you ask it in. That's not a feature you can toggle off. It's a fundamental property of how these models learn from the web.

The study doesn't claim AI companies are colluding with governments. It claims something more mundane and more difficult to fix: the web itself is not neutral, and AI models learn from the web as it exists, not as we wish it existed.

Whether users actually care about this distinction remains the real question. Most people just want their chatbot to work. They don't want to think about the institutional fingerprints embedded in every response. But those fingerprints are there, and they're measurable.

Time will tell if this matters enough to change how companies build AI systems. For now, the models keep learning from whatever the web gives them, and the web keeps reflecting whatever power shapes it. That's the actual story here, not the one about AI taking over the world.

Artūras Malašauskas is an AI Systems Integrator with 20+ years of production-grade web engineering experience. He has designed, shipped, and scaled enterprise Python/PHP systems for logistics, SaaS, and public-sector clients. For the past year, he has focused exclusively on AI integrations: deploying open-source LLMs, building generative media pipelines (image, audio, video), and engineering multi-agent workflows for real production environments. His standard: reproducibility, security, cost-efficient inference—no vaporware. He documents and evaluates emerging AI tooling, separating verified capabilities from marketing noise. Technical editor at: muza-ai.eu, ai-verslas.lt, ai-naujinos.lt Connect on LinkedIn