Pretty close. It's a smooth map with smooth inverse. If you've chosen chosen coordinates, it will transform them for you.
I think most physicists probably have a reasonable intuitive idea what a diffeomorphism is, since most have probably learned something about relativity.