How is one supposed to know the hash without also having had access to the file in some way and at some point in time?

If someone else knows the hash of your super secret document they must have had access to your super secret document or you must have given them the hash.