Have you seen this interview with GPT-3, the newest AI model?
I always think how amazing it would/will be, once we have RPGs like Ultima where the NPCs are like this. It's only a matter of years it seems.
Games like Event0 and even Shroud of the Avatar explore advanced dialogue systems without AI, and I also think that even without AI, we could make a pretty decent one. The trick is to have a query system allowing you to ask any character about any topic at any time, similar to Ultima VI, but expanded and modernized for easy use even for controller users. And the characters will respond based on their expertise level on that topic.
I first got this idea for a query system while playing Breath of the Wild. A quest tells you to find someone whose name ends in "son" SOMEWHERE in the world. You should be able to ask around to hone in on a location - which would be both realistic and fun, but since you cannot, you have to randomly stumble around, talking to everyone, hoping that 50 hours later you'll just happen upon the right person - which is bad detective work in my opinion (and kind of how investigation in Ultima V happens - randomly explore until you hit the right chains of NPCs).
In BOTW that frustration made me realize that a detective investigation system should exist in every game as an OPTIONAL, "unnecessary" query system - a built-in hint system that feels fun to use, instead of feeling demeaning like many hint systems. But the game should not be built to require its use as a crutch, just like how so many modern games are unplayable if you turn off the GPS mini map hud, since the games were designed to require its waypoint and path markers to be visible, instead of making the quests diegetically explorable within the context of the world.
I saw a video about Morrowind's design (highlighted there so I can remember to embed it when I find it) that detailed how characters would give actual directions to quest objectives using landmarks and such, such that you would not need to be and feel like a cyborg terminator robot, just to be able to play through and reasonably complete a medieval fantasy game that's supposed to be about immersion.
I think there's untapped potential here in bringing that motivated questing, that diegesis, back to game design.
Here's the original fb UDIC post.