At the risk of over-simplifying, Transformers (what ChatGPT, Claude, etc. use), are like your phone's auto-complete sentence on steroids. Transformers don't understand the words they spit out, they simply output what sounds like what the next word could probably be.
For example, you and I both know that the sentence "I like apples." and "I like oranges" are two inherently different sentences because we know that an orange is fundamentally and materially different compared to an apple. We know that any thought related to oranges should be related to, say, mandarins or clementines. We know inherently that any thought related to apples should be like fuji, gala, or red delicious. Transformers "guess" what the next word is based on the probability of the next word being somewhat close to the idea of "apple" and "oranges" .
Apples and oranges should not be close to each other when we think about either of them separately.
However, a Transformer will see apples and oranges together within the "fruit" space. So it will look at "fruit" space and think "well, apples and oranges are together in the 'fruit' space, so I think the next word will be blueberries".
It's all just one big probability model instead of a model that has an inherent understanding of the material world. That is, Transformers are meant to
sound like they know what they're talking about, but the reality is that they do not have an inherent, material understanding of what they're talking about. It's why if you ask an AI a question about a field you are an expert in, you can spot the hallucinations easily because you have a fundamental understanding of what you're asking about. In contrast, ask an AI about a field you are unfamiliar with, you'll be met with a wall of text that only sounds convincing, but isn't based on anything material in the real world.
It's why AI "lawyers" are getting in trouble for citing laws that never exist.
And why Deloitte got in trouble for having its AI cite research that never existed in a government report.
And the way the models are trained, they're trained for engagement, not actual intelligence. It's why ChatGPT will always say "You're so smart!" even when you give it the dumbest idea in the world.