No. An invalid pointer can point to literally anything. Just because the pointer type says it points to T doesn't mean it really does.

With an index into Vec<T> you are at least guaranteed that the object will be a T.

You are guaranteed memory safety, but you aren't guaranteed to have a correct reference.