We’re seeing an increasing number of news stories about people asking generative AI chat bots for advice, following that advice, and disaster ensuing. I am here to tell you: do not (uncritically) ask AI for instructions on how to do something. Do not treat AI like a search engine. Generative AI as it exists right now is not the correct tool for these uses. I promise I am not just being an AI hater here. There are a lot of good reasons to dislike the major generative AI companies: they’re telling us they’re going to take all our jobs, they’re taking up land and resources to build data centers, and their CEOs are annoying. But I think there’s one thing we’ve broadly lost sight of in this current era of AI apocalypse: the training data.

I am a scientist, I am old enough (read: not that old) to remember when we called AI “machine learning.” A common example in my field is using machine learning for genomic prediction. We would like to be able to take a small piece of leaf from baby plants, sequence the DNA therein, and have a reasonable idea of the quality of the fruit the plant will eventually produce. This would save us a lot of time and money, especially in tree fruit land where we have to wait years to see what quality fruit a tree will produce. In the breeding programs that are using genomic selection, it’s already saving them a lot of time and money. At every single scientific talk I have ever attended that covered genomic prediction and/or machine learning, one thing was made extremely clear: the training data is everything. The shorthand we use for this concept is “Garbage in, garbage out.”

Training data is what it sounds like: it’s the data you feed to an AI/machine learning model to teach it how to do its job. When we want an algorithm to be able to perform genomic prediction, we give it thousands of known scenarios: genetic information and physical trait (phenotypic) information for thousands of individual plants. The idea is that by training your model on these known scenarios, you can give it an unknown scenario (genetic information with no phenotypic information) and it will accurately predict the unknowns. With genetic data on plants, I personally would like to see a trait prediction accuracy above ~70% to feel like the software is worth my time and effort. If it’s not much better than a coin flip, I will save myself the trouble of using the software. With something like cancer screening, you want as close to 100% prediction accuracy as you can get, although 100% accuracy is not usually a result that comes up in real-life science. It’s how it goes.

If your data collection is sloppy and imprecise, the training data you feed your algorithm will be sloppy and imprecise, and your end model will be sloppy and imprecise. Setting up genomic prediction models is highly labor-intensive, because doing this exercise half-assed will result in a half-assed model, and why bother? We want to be able to predict fruit size, so we are going to measure the diameter of thousands of fruit off thousands of trees. We want to be able to predict sugar content, so we are going to measure sugar content in thousands of fruit off of thousands of trees. Do it right the first time, and the payoff (being able to sequence young seedlings and get an idea of future fruit quality) will be well worth the labor.

What do we know about the training data that ChatGPT, Claude, Grok, and their brethren received? We know they scraped, among other things, the entire public internet. Uncounted billions of pages of information posted online, plus every book the companies can get their filthy little hands on. Reddit started officially licensing the content of its forums to AI companies in 2024, but automated web-trawling scrapers probably sucked up most of Reddit well before any of it was officially handed over. The same with stackoverflow, twitter, old classic car forums, fanfiction websites, anything else you can think of on the open internet.

All of these generative AI models are Large Language Models (LLMs), so what they’re doing with their training data is evaluating the probability of individual words going into a sentence together. That’s it, that’s all. The word “banana” could be followed by “tree” or “pancakes” or “split,” commonly, but is unlikely to be followed by the word “tire.” If you want to build software that will make sentences that sound relatively human, feeding an LLM the whole internet and thousands of books is a good way to do that. If you want software that will give you good advice on gardening, these LLMs are not it. Because the training data includes every confident dumbass who ever posted about growing an exponentially-increasing number of tomatoes to achieve wealth. The training data includes every sarcastic remark from a well-informed person, but the training data does not include any note that the remark was sarcastic (remember the Elmer’s glue on pizza advice google was giving a while back?). The training data includes every earnest but incorrect answer to that question you’re asking.

The LLMs have every statement ever typed into Al Gore’s Internet, but they do not have any way to evaluate each statement as true or false, just more probable or less probable. You could add that information to the training data, by the way. It’s not impossible, you just need human beings to manually go through every single bit of training data for the LLMs and give each statement a value of true or false. Easy, right? To go through the whole internet and assign every single sentence as correct or not? Shouldn’t take long. Don’t forget the books, too, goodness knows there have been a lot of false statements published in books over the years. If that were how generative AI was being trained, with added encoding of what’s true, what’s false, what’s sarcasm, what’s more nuanced than the writer is letting on, I would be much less exhausted with the whole enterprise. But the companies making LLMs are not going to do that, because it would be wildly expensive and take years, if not decades. And they want their money now. They want your money now. So they’re giving us models that will respond to our questions with the sequence of words that seems the be the most probable answer*, and some of us are giving ourselves bromine poisoning as a result.

*this is an oversimplification, and also not taking into account the sycophancy encoded into LLMs to encourage you to keep using them because they make you feel good. That’s a whole other blog post.