Predatory journals contaminate LLM training data with fake science
Large language models are increasingly ingesting content from predatory journals, causing them to present fabricated scientific findings as fact. Experts warn this manipulation risk necessitates new systems for weighting scientific credibility in training datasets.
Till Krause, Professor for Media and Communication at Hochschule Landshut, argues that publications from predatory journals are increasingly flowing into the training data of large language models. This contamination causes models to present fabricated scientific findings as fact because they cannot reliably distinguish between genuine peer-reviewed papers and predatory journal content during training. A Swedish researcher conducted an experiment where she invented a non-existent disease and published fake papers about it. When querying LLMs about this condition, the models displayed the fake disease as real. Krause warns that this manipulation risk is immense, potentially allowing false claims on topics like climate change or tobacco to be validated by AI. He notes that even frontier model developers may lack the expertise to filter out predatory sources. Krause advocates for ranking and weighting scientific sources by truthfulness and seriousness in training datasets, similar to how Reddit content is weighted seven times in recent ChatGPT training sets.