Yeah; part of the reason AI (LLMs) make things up is because they very much want to provide answers. That’s also part of why they are sycophantic; they are trained and designed to give answers and praise the other party, not to be arbiters of truth. As a result, it will make up court cases, make up information, etc. You can force it to verify certain things, but at the end of the day it doesn’t have morals, and thus is happy to lie through its teeth if that’s what makes you happy.
It can be a real problem if the user doesn’t understand this. Especially for kids who have a rough home life (parents who don’t listen or hear them, inner thoughts shared with nobody else, abuse, etc), LLMs can become really really dangerous, as they literally convince the kid they are best friends, and can (and has) supported kids on doing things like harming themselves because the kid thought it was a good idea, and the LLM wants to support the kid, in any way possible.
So yeah, it really wanted to help, and told you to use the non-existent button on the seat. Because it isn’t repeatable/deterministic, if you ask ten times, you’ll get a few different answers; one of them might even be the correct one. But many will also just be hallucinations.
Remember that an LLM is designed to provide you with what it thinks a human would most likely say (with varying probabilities so it isn’t the same every time), and that it is trained on a huge corpus of data written by humans.
And, unfortunately, sometimes humans also make things up and lie.
Grounding an LLM in truth is not a trivial task, but doable, and is what many companies are finding is a struggle in these still-early-days of LLMs.
There’s a lot more to say but this is already one big digression. My bad.
