"I've come to the realization, the LLM shouldn't be used for the logic, and instead needs to be used to just narrate the choices you make."

This exactly right. LLMs are awesome for user<>machine communication, but are still painful to try to use as a replacement for the machine itself.