This way of thinking is accurate. And hyping LLMs to be a precursor to AGI is actually the unhelpful thing, IMHO.

I recommend you look a bit at the work Emily M. Bender is doing. She’s a computational linguist and doesn’t have much good to say about the “Sparks of AGI” paper.

why are they so sure that we’re not just doing the same trick?

Because: Even if we don’t know what makes up conscience, we DO know a fair bit of how language works. And LLMs can mimic form, but lack some semblance of intentionality. Again, Emily M. Bender can summize this better as I could.