Yes. This is correct. Once, I had misconfigured my mail server when adding another domain I had bought and it rejected an e-mail. While I was looking into it, I saw in the logs that the sending party had retried after 15 minutes so I corrected my config and waited another 15 and I got the e-mail. Of course, number of retries, if any, and time between attempts will vary between different servers and their configs.
Another story I have about e-mail is that another misconfiguration on my part caused mail from me to others to be spam filtered or outright dropped by some receivers. It wasn't until I attempted to send an e-mail to a Debian mailing list that I learned why that was. Their e-mail server was nice enough to reply with the exact reason why they would not accept my mail. I was then able to fix it. (Though I don't remember if they did so in the form of what was said over SMTP and seen by me in logs, or if they bounced the e-mail.)
Another story I have about e-mail is that another misconfiguration on my part caused mail from me to others to be spam filtered or outright dropped by some receivers. It wasn't until I attempted to send an e-mail to a Debian mailing list that I learned why that was. Their e-mail server was nice enough to reply with the exact reason why they would not accept my mail. I was then able to fix it. (Though I don't remember if they did so in the form of what was said over SMTP and seen by me in logs, or if they bounced the e-mail.)