> I would often get put in directories that I didn't want.

I solved this by combining it with fzf. Get all the directories you've ever visited and pass on to fzf (sorted by frequency). Then do your matching. You can trivially see if the match is taking you where you want. If not it is likely the second or third match. You're no longer constrained to navigating only to the top matched directory.

I have this bound to a keystroke in my shell.