Compare chatgpt to, like, Watson (which is like a wooden duck that has a cadbury creme egg machine in it -- a prolog-like formal inference engine).
Not that you should trust Watson, but at least it *theoretically* can *try* to check whether or not the things it says might be true.
Purely statistical models are purely system 1, so even if one somehow got human-level intelligence, the form of that intelligence would be like an extremely drunk & sleep-deprived person with severe untreated ADHD.